Search code examples
apache-sparkapache-spark-standalone

Spark - local standalone mode won't write to history server


I'm trying to enable Spark history server in single standalone mode on my Mac. I have a spark-master service running and am able to run jobs. I also have a history-server service running on localhost. I'm able to view it in my browser but there are no records of any applications on it. I can only assume I've missed some configuration bur I don't know what. Can anybody please advise how to configure history server to work with local standalone Spark?

Here is my configuraiton:

# spark.master spark://master:7077 
spark.eventLog.enabled true 
# spark.eventLog.dir hdfs://namenode:8021/directory 
# spark.serializer org.apache.spark.serializer.KryoSerializer 
# spark.driver.memory 5g 
# spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

Solution

  • Directory spark-events was spelled incorrectly. I had spark-event. Once I renamed it I was able to view the job logs.