Here is my sbt for spark with scala on Intellij
version := "0.1"
scalaVersion := "2.11.11"
// https://mvnrepository.com/artifact/org.apache.spark/spark-graphx_2.10
libraryDependencies += "org.apache.spark" % "spark-graphx_2.10" % "2.1.0"
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.11
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
I get the following error
[error] (*:update) Conflicting cross-version suffixes in:org.apache.spark:spark-launcher, org.json4s:json4s-ast, org.apache.spark:spark-network-shuffle, org.scalatest:scalatest, com.twitter:chill, org.json4s:json4s-jackson, com.fasterxml.jackson.module:jackson-module-scala, org.json4s:json4s-core,org.apache.spark:spark-unsafe, org.apache.spark:spark-tags, org.apache.spark:spark-core, org.apache.spark:spark-network-common
[error] (*:ssExtractDependencies) Conflicting cross-version suffixes in: org.apache.spark:spark-launcher, org.json4s:json4s-ast, org.apache.spark:spark-network-shuffle, org.scalatest:scalatest, com.twitter:chill, org.json4s:json4s-jackson, com.fasterxml.jackson.module:jackson-module-scala, org.json4s:json4s-core, org.apache.spark:spark-unsafe, org.apache.spark:spark-tags, org.apache.spark:spark-core, org.apache.spark:spark-network-common
Can anyone let me know how to correct this. To me it looks like spark-graphx is contradicting with spark_core. How do I find the correct SBT file? or what should i do to get this sbt file working. I am ok with lowering the version of graphx as well as spark but would prefer to lower the version of spark.
In you sbt file, the dependencies are called spark-_. You're using different scala version in your graphX dependency.
If you use Spark 2.x I recommend use scala 2.11 so you only have to change graphx dependency.
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0"
Anyway, you don't need to write scala version in every dependency. Sbt will infer according scalaVersion value if you write two percentage sign %%. Here, you can see an example.