Search code examples
javamongodbapache-flink

MongoDB as datasource to Flink

Can MongoDB be used as a datasource to Apache Flink for processing the Streaming Data?

What is the native implementation of Apache Flink to use No-SQL Database as data source?

Solution

  • Currently, Flink does not have a dedicated connector to read from MongoDB. What you can do is the following:

    • Use StreamExecutionEnvironment.createInput and provide a Hadoop input format for MongoDB using Flink's wrapper input format
    • Implement your own MongoDB source via implementing SourceFunction/ParallelSourceFunction

    The former should give you at-least-once processing guarantees since the MongoDB collection is completely re-read in case of a recovery. Depending on the functionality of the MongoDB client, you might be able to implement exactly-once processing guarantees with the latter approach.