Search code examples
dockerapache-flinkflink-streaming

How to get TaskManager containers registered with Job Manager container for flink?

Trying to play with flink docker image for the first time. I am following instructions at https://hub.docker.com/_/flink which says

You can run a JobManager (master).

$ docker run --name flink_jobmanager -d -t flink jobmanager

You can also run a TaskManager (worker). Notice that workers need to register with the JobManager directly or via ZooKeeper so the master starts to send them tasks to execute.

$ docker run --name flink_taskmanager -d -t flink taskmanager

Can someone explain how does this taskmanager register with the jobmanager from these commands?

Thanks

Solution

  • In order to start a Flink cluster on Docker I would strongly recommend to use docker-compose whose config file you can also find here.

    If you want to set up a Flink cluster using docker manually, then you have to start the containers so that they can resolve their names. First you need to create a custom network via

    docker network create my-network

    Next, you have to start the jobmanager with this network and configure the name as well as the hostname to be same. That way Flink will bind to the hostname which is resolvable.

    docker run --name jobmanager --hostname jobmanager --rm --net my-network -d -t flink jobmanager

    Last but not least, we need to start the taskmanager and tell him the name of the JobManager. This is done by setting the environment variable JOB_MANAGER_RPC_ADDRESS to `jobmanager.

    docker run --name taskmanager --net my-network -e JOB_MANAGER_RPC_ADDRESS=jobmanager -t -d flink taskmanager