"Delegation Token can be issued only with kerberos or web authentication" when restarting spark context after idle

Question

I try to deploy spark application to kerberized hadoop cluster which is controlled by YARN. Version of Spark is 1.5.0-cdh5.5.2.

I'm facing strange exception, when stopping SparkContext after more than 10 seconds idle and initializating a new one.

I've tried to do something similar to what did this developer, and explicitly specified hdfs namenode address, but it didn't help.

What is more confusing that everything works fine if I don't reset SparkContext at all or reset it in less than ~10 seconds after last command was executed in this spark context.

How could I fix it?

Here is minimized case where problem is met:

package demo;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;


public class App
{
    public static void main( String[] args ) throws Exception {

        SparkConf sparkConf = new SparkConf();
        sparkConf.setAppName("demo");
        sparkConf.set("spark.yarn.access.namenodes", "hdfs://hdp:8020");

        JavaSparkContext jsc = new JavaSparkContext(sparkConf);

        int waiting_time = 10;
        System.out.println("Waiting time: " + Integer.toString(waiting_time));
        Thread.sleep(waiting_time * 1000);

        jsc.stop();

        jsc = new JavaSparkContext(sparkConf); // "Delegation token ..." exception here
    }

}

Stack trace when exception is raised: https://gist.github.com/anonymous/18e15010010069b119aa0934d6f42726

spark-submit command:

spark-submit --principal mp@LAGOON --keytab mp.keytab --master yarn-client --class demo.App demo.jar

Answer 1

Problem was caused by this problem: https://issues.apache.org/jira/browse/SPARK-15754

In Spark 1.6.2 it was fixed.