Search code examples
apache-sparkspark-graphx

How to measure execution time of Spark GraphX application?

What is the most reliable way to find the total execution time of a program using Graphx , is it best to use the history server or in the program use variable with System.currentTimeMillis() or something else?

Solution

  • Measuring execution time for a Spark GraphX application is no different than measuring performance of any other Spark application.

    I'd strongly recommend using web UI while the Spark application is up and running and Spark History Server afterwards.

    You can also use SparkListener to build something similar to web UI (after all web UI is just a bunch of SparkListeners).

    System.currentTimeMillis() would work, too, but why reinventing the wheel if almost all (if not all?) is already available in Spark?