Set Cloudera application tags for Spark application

Question

I have set spark.yarn.tags in my spark application and it is visible as well in my config when printed. But Cloudera manager is unable to detect it in application_tags field of yarn application.

Does application_tags map to spark.yarn.tags for spark applications?

Answer 1

I think I found the solution. When spark.yarn.tags is set while calling spark-submit, cloudera manager detects it. So I believe it is something it requires before spark context is created, hence it has to be passed as conf while submitting.

This is how it can be passed to the spark-submit --conf spark.yarn.tags=tag-name