Search code examples
apache-flinkinfluxdbflink-streaming

Is there exist some good open-source connectors that achieve the data transmission from InfluxDB to Flink


my team encounters some problems when we try to construct our own temporal database system. At every begin, we try to understand the process of data transmission between influx DB and Flink. Our underlying idea is: Flink read the data from the influxDB, and Flink uses these data to conduct some modeling based on ML library of Flink. (As the red circle in the Picture). However, we only find some open-source repositories that provide a sink that can send data to InfluxDB from Flink (Such as Flink InfluxDB Connector). Therefore, if you know some good examples like what I describe, please tell me. Thanks a lot.

enter image description here


Solution

  • If your goal is to stream data from InfluxDB, I would look at Kapacitor. As far as I know there are no current source functions for InfluxDB. Also, to be frank, it's unlikely InfluxDB can serve the kind of data volume that would drive you to use a distributed processing system like Flink, as it's optimized for high write speed and fast queries that target specific data.