Where is yarn.nodemanager.log-dirs in spark?

Question

I have looked into:

• log4j2.properties in /etc/spark/conf
• yarn-site.xml
• yarn-env.sh (via YARN_LOG_DIR it is not getting set. In fact while running a job there is no env variable YARN_LOG_DIR in my executors)
• log4j.properties in /etc/hadoop/conf

Where can I find and modify the yarn.nodemanager.log-dirs property?

Answer 1

To find this, we need to traverse some of Hadoop's source code:

• yarn.nodemanager.log-dirs defaults to ${yarn.log.dir}/userlogs.
• yarn.log.dir defaults to $HADOOP_LOG_DIR
• $HADOOP_LOG_DIR defaults to ${HADOOP_HOME}/logs

So, have a look at $HADOOP_HOME/logs/userlogs to see whether you find something in there!

If you want to edit it, you can do either of the following:

• edit $HADOOP_HOME
• edit $HADOOP_LOG_DIR
• add -Dyarn.log.dir=<your_chosen_value> to your spark application
• add -Dyarn.nodemanager.log-dirs=<your_chosen_value> to your spark application