I submitted a job using spark-submit, the job finished correctly but it does not show in Yarn.
how can I view the process directly in Yarn ? I already specified with yarn client mode, but in yarn web console I see nothing.
Based on this other Stack Overflow thread, yarn application -list -appStates FINISHED might do the job. Also in this same thread, you will find another possible reason:
It turns out that I had enabled Log aggregation in YARN but had set the yarn.nodemanager.remote-app-log-dir to a custom hdfs directory (/tmp/yarnlogs), So logs were actually getting aggregated at /tmp/yarnlogs in HDFS, but the yarn command was still searching for logs at the default location on HDFS (/tmp/logs). So changing the property to its default value fixed it for me.
Hope it helps!