Bigtable error with sbt assembly fat JAR (Neither Jetty ALPN nor OpenSSL are available)

Question

I would like to build a Restful API with akka-http able to retrieve data from Bigtable (HBase).

The Bigtable client API requires netty-tcnative-boringssl-static to connect. This works pretty well inside my Intellij IDE, but when I build a fat JAR with sbt-assembly, and then run the server, I get the following error:

2017-01-10 12:03:41 ERROR BigtableSession:129 - Neither Jetty ALPN nor OpenSSL are available. OpenSSL unavailability cause:
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora, netty-tcnative]
Exception in thread "main" java.lang.ExceptionInInitializerError
        at mycompany.algo.serving.model.algoServingModelLoaderTest$.loadLastModel(algoServingModelLoaderTest.scala:36)
        at mycompany.algo.serving.algoServingLauncherTest$$anonfun$4.apply(algoServingLauncherTest.scala:38)
        at mycompany.algo.serving.algoServingLauncherTest$$anonfun$4.apply(algoServingLauncherTest.scala:38)
        at mycompany.serving.MultiPredictorEbapServing$$anonfun$loadPredictors$1.apply(MultiPredictorEbapServing.scala:25)
        at mycompany.serving.MultiPredictorEbapServing$$anonfun$loadPredictors$1.apply(MultiPredictorEbapServing.scala:25)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at mycompany.serving.MultiPredictorEbapServing.loadPredictors(MultiPredictorEbapServing.scala:25)
        at mycompany.algo.serving.algoServingLauncherTest$.delayedEndpoint$mycompany$algo$serving$algoServingLauncherTest$1(algoServingLauncherTest.scala:38)
        at mycompany.algo.serving.algoServingLauncherTest$delayedInit$body.apply(algoServingLauncherTest.scala:11)
        at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
        at scala.App$$anonfun$main$1.apply(App.scala:76)
        at scala.App$$anonfun$main$1.apply(App.scala:76)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
        at scala.App$class.main(App.scala:76)
        at mycompany.algo.serving.algoServingLauncherTest$.main(algoServingLauncherTest.scala:11)
        at mycompany.algo.serving.algoServingLauncherTest.main(algoServingLauncherTest.scala)
Caused by: java.lang.IllegalgotateException: Could not find an appropriate constructor for com.google.cloud.bigtable.hbase1_2.BigtableConnection
        at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:88)
        at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:72)
        at mycompany.algo.serving.model.algoServingModel$.<init>(algoServingModel.scala:25)
        at mycompany.algo.serving.model.algoServingModel$.<clinit>(algoServingModel.scala)
        ... 18 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:85)
        ... 21 more
Caused by: java.lang.IllegalgotateException: Neither Jetty ALPN nor OpenSSL via netty-tcnative were properly configured.
        at com.google.cloud.bigtable.grpc.BigtableSession.<init>(BigtableSession.java:279)
        at org.apache.hadoop.hbase.client.AbstractBigtableConnection.<init>(AbstractBigtableConnection.java:137)
        at org.apache.hadoop.hbase.client.AbstractBigtableConnection.<init>(AbstractBigtableConnection.java:104)
        at com.google.cloud.bigtable.hbase1_2.BigtableConnection.<init>(BigtableConnection.java:50)
        ... 26 more

What should I do ?

I am using Scala 2.11.8 and my sbt looks like:

dependencies ++= Seq(
  "com.google.cloud" % "google-cloud" % "0.7.0",
  "com.google.cloud.bigtable" % "bigtable-hbase-1.2" % "0.9.4",
  "io.netty" % "netty-tcnative-boringssl-static" % "1.1.33.Fork19",
  "org.apache.hbase" % "hbase-server" % "1.2.1",
  "org.apache.hbase" % "hbase-client" % "1.2.1",
  "org.apache.hbase" % "hbase-common" % "1.2.1",
)

Thanks for your help

EDIT Edited since I originally though it was caused by akka, but could reproduice it without Akka

EDIT My bad, the native libraries were evicted with my sbt assembly merge strategy

EDIT This works with Maven, the issue seems to be with sbt

SOLUTION This was caused by sbt-assembly default merge strategy. I needed to add the below settings

val settings = Seq(
    assemblyMergeStrategy in assembly := {
      case PathList("META-INF", xs @ _*) =>
        xs map {_.toLowerCase} match {
          case "native" :: xs =>
            MergeStrategy.singleOrError
          case _ => MergeStrategy.discard
        }
      case "reference.conf" => MergeStrategy.concat
      case x => MergeStrategy.first
    }
)

Answer 1

The problem was with SBT merge strategy, I used this strategy to solve the problem.

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
  case _ => MergeStrategy.first
}