Search code examples
apache-flinkflink-streaming

How to write flink job output to a single location instead of two locations


Currently, I have Flink operating as a cluster through Docker, employing three containers: one for the jobmanager and two for the taskmanagers. Whenever I submit a job, the job's output gets written to both containers. I am interested in finding a solution to consolidate the output in a single location, preferably either on the local machine. Is there a way to achieve this?


Solution

  • You could:

    • use a single task manager with 2x as many slots (you'll still end up with multiple files, but they'll be in the same directory on the same machine)
    • reduce the parallelism of the job
    • reduce the parallelism of the sink