SparkContext.setLogLevel("DEBUG") doesn't works in Cluster

Question

I'm trying to control my Spark logs use sc.setLogLevel("ERROR"); seems like it doesn't work in the cluster environment. Can anyone help?

public static JavaSparkContext getSparkContext(String appName, SparkConf conf) {
    SparkSession spark = getSparkSession(appName, conf);
    JavaSparkContext sc = new JavaSparkContext(spark.sparkContext());
    sc.setLogLevel("WARN");
    return sc;
}

Answer 1

To configure log levels, add the following options to your spark submit command:

'--conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=custom-log4j.properties"'

'--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=custom-log4j.properties"'

This assumes you have a file called custom-log4j.properties on the classpath. This log4j can then control the verbosity of spark's logging.