how to download dse.jar

Question

I am trying to use DataStax Enterprise 4.6 to write a Spark application in Java and run it in DSE's Spark analytics mode.

The code for creating a Spark context using DSEConfHelper:

SparkConf conf = DseSparkConfHelper.enrichSparkConf(new SparkConf())
            .setAppName( "My application");

To use DSEConfHelper we need to import com.datastax.bdp.spark.DseSparkConfHelper which is located in dse.jar.

In my pom.xml I have included the dependency:

<dependency>
    <groupId>com.datastax</groupId>
    <artifactId>bdp</artifactId>
    <version>4.6.0</version>
</dependency>

But Maven cannot download dse.jar.

Please help me.

The reference for code for creating a Spark context is taken from: http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/spark/sparkJavaApi.html

Answer 1

Edit: This has been entirely superceded by the com.datastax.dse.dse-spark-dependencies artifact. Add it to your pom.xml:

<dependencies>
  <dependency>
    <groupId>com.datastax.dse</groupId>
    <artifactId>dse-spark-dependencies</artifactId>
    <version>${dse.version}</version>
    <scope>provided</scope>
  </dependency>
<dependencies>

<repositories>
  <repository>
    <id>DataStax-Repo</id>
    <url>https://repo.datastax.com/public-repos/</url>
  </repository>
</repositories>

See https://github.com/datastax/SparkBuildExamples for Maven, SBT, and Gradle example projects.

Original, outdated answer:

You have to manually install dse.jar as of right now. There are two ways of doing this.

Option 1 Install the JAR file using mvn install:

$ mvn install:install-file -Dfile=<path-to-dse.jar> -DgroupId=com.datastax -DartficactId=bdp -Dversion=4.6.0

Option 2 Manually copy dse.jar from your install location to ${project.basedir}/lib/. Then modify your pom.xml:

<dependency>
  <groupId>com.datastax</groupId>
  <artifactId>bdp</artifactId>
  <version>4.6.0</version>
  <scope>system</scope>
  <systemPath>${project.basedir}/lib/dse.jar</systemPath>
</dependency>

I don't really know why you're calling the artifact "bdp", but for these purposes it doesn't matter, and I just used it as well.