Search code examples
apache-sparkpysparkpycharm

How to open spark web ui while running pyspark code in pycharm?

I am running pyspark program in pycharm local on windows 10 machine . I want to open spark web ui to monitor job and understand metrics showed over spark web ui . While running same code on jupyter I can access web ui but when I run program using pycharm I do not see option to access spark web ui , even I tried url local:4040 or localhost:4041 but , this did not work .

Solution

  • You can configure which port that ui can access in sparksession.

    Example:

    spark = SparkSession.builder.master("local").\
appName("Word Count").\
config("spark.driver.bindAddress","localhost").\
config("spark.ui.port","4050").\
getOrCreate()

    Now once the spark session initialized then we can access spark ui in http://localhost:4050/jobs/ until the pycharm job finished executing.

    In case if the port is not able to access at the time then in your log spark prints where does we can access the spark ui.

    Sample log:

    2020-08-17 14:39:37 WARN Utils:66 - Service 'SparkUI' could not bind on port 4050. Attempting port 4051.