Search code examples
javaapache-sparkspark-jobserver

Persisting Spark Jobserver NamedObjects using Java

Using Spark Jobserver 0.6.2 and Apache Spark 2.0.2, I have already some functionalities implemented. But I can't find how to persist a Dataset to be shared across diferents jobs on a specific context in java. It's this functionality only available for Scala? Because on internet I only can find Scala examples. I have check on java and there are some NamedObject* classes in the job-server-api_2.11.jar but not idea how can I use them. Actually my Job classes extends from spark.jobserver.JavaSparkJob.

Thanks.

Solution

    1. You can't use Job server 0.6.2 with Spark 2.0.2. See the compatibility matrix in README.md (https://github.com/spark-jobserver/spark-jobserver#version-information)

    2. You might have to use spark-2.0-preview branch.

    3. Here is a thread which talks about new NamedObjects API in java. One of the thread has latest example using new API as attachment too https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/spark-jobserver/lFpuiQp4toU/xkZtkxTHBAAJ