Search code examples
hadoop-yarnapache-flink

flink web.port can not be configured correctly in yarn mode

I want to get flink's metrics information via REST api, my flink is managed by YARN, but after changing web.port configuration in flink-conf.yaml, the change has no affect, and the web.port in the flink dashboard is always 0. So I can not get the flink metrics information via REST api.

enter image description here

Environment: ubuntu 16.04 openjdk-8 hadoop 2.7.1.2.3.6.0-3796 flink 1.4.0

Solution

  • When running Flink on Yarn, Flink will pick a random port (0) for the web UI in order to avoid port conflicts with other applications running on the same machine.

    In order to access the Flink web UI you can query the Yarn web application proxy (YarnResourceManagerURL/proxy/application_/...). But be aware that only GET requests are properly forwarded to the Yarn application.

    Alternatively, Flink logs the web UI URL to stdout when starting a Yarn session. Moreover, you can retrieve the chosen port from the log files. In newer versions (>= 1.5) Flink will log Rest endpoint listening at hostname:port on INFO level and in older versions (<= 1.4 or if using the legacy mode) Flink will log Web frontend listening at hostname:port.