Can somebody tell me how to build the Spark-Cassandra Connector assembly? I've tried following the instructions on the Github page https://github.com/datastax/spark-cassandra-connector but I just get hundreds of "deduplicate" errors.
I'm using Scala 2.11.7 with Spark 1.5.1 (which I built for Scala 2.11) and SBT 13.8.
I did the following:
git clone https://github.com/datastax/spark-cassandra-connector.git
cd spark-cassandra-connector/
sbt -Dscala-2.11=true assembly
The build process runs for a while, but then starts spitting out hundreds of "deduplicate" errors and fails. I have no idea where to start fixing this, but right now as far as I can tell the assembly build process for this project just doesn't work.
Any tips on how I can fix this?
It appears the build for 2.11 is broken and you should report it to the project. Dunno how to fix it right now.
➜ spark-cassandra-connector git:(master) sbt
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Loading project definition from /Users/jacek/dev/oss/spark-cassandra-connector/project
Using releases: https://oss.sonatype.org/service/local/staging/deploy/maven2 for releases
Using snapshots: https://oss.sonatype.org/content/repositories/snapshots for snapshots
Scala: 2.10.5 [To build against Scala 2.11 use '-Dscala-2.11=true']
Scala Binary: 2.10
Java: target=1.7 user=1.8.0_66
[info] Set current project to root (in build file:/Users/jacek/dev/oss/spark-cassandra-connector/)
[root]> update
...
[info] Done updating.
[info] Done updating.
[success] Total time: 314 s, completed Dec 2, 2015 10:26:01 AM
[root]>
➜ spark-cassandra-connector git:(master) sbt -Dscala-2.11=true
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Loading project definition from /Users/jacek/dev/oss/spark-cassandra-connector/project
Using releases: https://oss.sonatype.org/service/local/staging/deploy/maven2 for releases
Using snapshots: https://oss.sonatype.org/content/repositories/snapshots for snapshots
Scala: 2.11.7
Scala Binary: 2.11
Java: target=1.7 user=1.8.0_66
[info] Set current project to root (in build file:/Users/jacek/dev/oss/spark-cassandra-connector/)
[root]> update
...
[error] impossible to get artifacts when data has not been loaded. IvyNode = org.slf4j#slf4j-log4j12;1.7.6
...
[trace] Stack trace suppressed: run last spark-cassandra-connector-embedded/*:update for the full output.
[error] (spark-cassandra-connector-embedded/*:update) java.lang.IllegalStateException: impossible to get artifacts when data has not been loaded. IvyNode = org.slf4j#slf4j-log4j12;1.7.6
[error] Total time: 9 s, completed Dec 2, 2015 10:27:19 AM
I filed an issue https://datastax-oss.atlassian.net/browse/SPARKC-295.