Apache spark Web UI on Amazon EMR not working

Question

We are using Spark on Amazon EMR 4.1. When accessing application master link on yarn resource manager we are seeing a blank page. Using Firefox debugging, we noticed that we got a HTTP 500 error. We have tried configuring proxy settings for AWS and also replacing the URL's internal hostname with external hostname/ip address but are still not able to access the web ui.

The same steps worked for us in the past on EMR 3.x version.

Answer 1

The problem seems like your proxy, I had the same problem Add this sbt repository or maven, whatever you use

SBT

"org.apache.hadoop" % "hadoop-yarn-server-web-proxy"

Maven

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-yarn-server-web-proxy</artifactId>
    <version>2.6.2</version>
</dependency>

its working for me