Search code examples
apache-sparkhadoop-yarnemramazon-emr

EMR 5.13: Spark 2.3.0 UI shows Executors remain alive

Ever since I've upgraded to EMR 5.13, I've been seeing strange metrics on the Spark & YARN UIs.

In this particular instance:

  • YARN showed that process completed YARN reports task completed
  • Ganglia shows that cluster has been idle since completion of last (118th) job
  • Spark UI also tells that all my 118 tasks have been completed All 118 jobs were completed

Even so, Spark UI reports that all Executors are alive, long (over 1 hr at time of writing) after the last job was completed. Executors are still alive with active tasks

Could this be a UI glitch or there's something else going on?

Frameworks / Platform:

  • EMR 5.13
  • Spark 2.3.0
  • Hive 2.3.2
  • Hadoop: Amazon 2.8.3
Solution

  • One executor with active tasks in your screen is marked as Dead. It shows statistics at the moment of termination.

    As you can see, executor #5 has processed 624 tasks before termination. Then yarn started a new executor #9 instead that completed 76 tasks