Search code examples
apache-flink

What does mean processing in apache flink?


I'm having trouble with the concept of processing in apache Flink. When it comes to processing in apache Flink, what does this process involve? Is there an example for that?


Solution

  • Flink ships with a bunch of examples, and there several more complete examples in the online training.

    The basic architecture involves a cluster of stateful processing nodes -- which you might think of, to a first approximation, as a shared key-value store. Live data streams are passing through these nodes. The state held in the cluster is being updated by your business logic in response to these data flows, and their output is being affected by this state.

    Typical use cases: ETL, fraud/anomaly detection, ride sharing platforms, logistics management.

    Flink can operate at very large scale, with low latency and high throughput. A talk like this one about dynamic pricing at Lyft should help get across the big ideas in the context of a real application: https://www.ververica.com/resources/flink-forward-san-francisco-2019/streaming-your-lyft-ride-prices