Search code examples
apache-flinkram

Why does Apache Flink need much RAM


I'm considering to use Apache Flink to process some stream data in my project.

However, I was told that Flink may need much RAM by a friend. Also, I've found something which told me the same thing: https://www.quora.com/What-is-the-difference-between-Apache-Flink-and-Apache-Spark

For now I haven't learnt a lot about Flink, I just succeeded in installing it and running the Word Count example.

So I'm wondering why Flink needs much RAM. What is the mainly reason? Some disadvantage of Flink itself? Or saving the historical data? or anything else?

Can I use something like Redis to avoid this issue?


Solution

  • That answer on Quora is rather old, and lacks specifics.

    It all depends on what you mean by "a lot of memory". I've seen Flink running on a cluster of Raspberry PIs -- see https://hal.inria.fr/hal-02463206/document. For another take on this, see also Extend Flink to edge computing with much lower footprint.

    The out-of-the-box configuration is designed to work pretty well across a wide set of use cases. So there is some room for optimization if you need to squeeze Flink down into a more resource-constrained environment.