Search code examples
javaapache-flinkflink-streaming

How to have one operator per vertex in Apache Flink?

I have a streaming job that I am running in Apache Flink.

It consists of the following operators:

  1. Source Function (generates data)
  2. Filter Function (filters out some data)
  3. GroupBy and aggregate (groups data based on a key and then runs an aggregation function over them)
  4. Another filter function
  5. File Sink (saves output to file)

However, when I sumbit the job I get the following graph. Query Graph - Flink UI

As you can see for example the 3rd and 4th steps are merged together. My question is, since I want to measure the throughput and latency of each step, is there a way to split each operator on its own vertex? In this example: Have a node for the groupBy-Aggregation and another one for the filter function (endsWith).

Thanks in advance! :)

Solution

  • You can do this with

    env.disableOperatorChaining();

    However, you should expect this to degrade performance, perhaps significantly. See Flink disableOperatorChaining Performance impact for more about this.