Search code examples
javascalaapache-sparksbtfluentd

object tools is not a member of package scala

In my spark application, I'm trying to use fluentd-scala-logger for which I had to include an additional dependency in my build.sbt These are the 2 lines I added in my build.sbt:

resolvers += "Apache Maven Central Repository" at "https://repo.maven.apache.org/maven2/"
"org.fluentd" %% "fluent-logger-scala" % "0.7.0"

My final build.sbt looks like this:

name := "sample"
version := "1.4"
scalaVersion := "2.11.8"
resolvers += "Apache Maven Central Repository" at "https://repo.maven.apache.org/maven2/"
libraryDependencies ++= Seq("org.elasticsearch" %% "elasticsearch-spark" % "2.1.2", "org.apache.spark" %% "spark-sql" % "2.1.2", "org.apache.kafka" % "kafka-clients" % "2.4.1", "org.fluentd" %% "fluent-logger-scala" % "0.7.0")

As soon as I do sbt package to bundle my spark app in a jar, I face below issue:

object tools is not a member of package scala
[error] import scala.tools.nsc.io.File

I didn't face this when my sbt earlier looked like this(without fluentd dependency):

name := "sample"
version := "1.4"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq("org.elasticsearch" %% "elasticsearch-spark" % "2.1.2", "org.apache.spark" %% "spark-sql" % "2.1.2", "org.apache.kafka" % "kafka-clients" % "2.4.1")

Is resolvers line causing the issue? or am I missing something else completely. I'm using sbt version 1.4.5 & Scala version: 2.11.8

Solution

  • I am not sure what causes it, but I found a solution. Please try to add the dependency like this:

    libraryDependencies += "org.fluentd" %% "fluent-logger-scala" % "0.7.0" intransitive()

    It will import this dependency without its dependencies.

    Having that said, I looked at this library dependencies, and tried to exclude all of them 1 by 1:

    libraryDependencies += "org.fluentd" %% "fluent-logger-scala" % "0.7.0" excludeAll(
  ExclusionRule("org.msgpack", "msgpack"),
  ExclusionRule("org.slf4j", "slf4j-api"),
  ExclusionRule("ch.qos.logback", "logback-classic"),
  ExclusionRule("junit", "junit"),
  )

    but it didn't work. So I really can't explain it.