Search code examples
apache-flinkapache-beamvisual-programming

is there a visual programming environment like Nifi or streamsets data collector for flink or beam-filnk


I am looking for a visual programming tool to abstract away coding when creating data stream flows to build CEP applications in Flink or Beam-Flink. Something like the Nifi or streamsets data collector graphical designers.

Is anyone aware of such a project?


Solution

  • Unfortunately, at the moment, there isn't such a visual pipeline construction tool readily available in the open source ecosystem.

    That said, this isn't an unsolvable problem -- in Apache Beam, it is tracked via BEAM-266, and it is conceivable that this becomes a part of the project in the medium term.

    Outside the open source, there are commercial tools built on top of Apache Beam that provide something like this, but in a somewhat reduced domain (e.g., data preparation) -- check out tools by Talend, Google Cloud, and others.