Search code examples
apache-flinkflink-cep

Flink CEP: Which method to join data streams for different type of events?

Suppose that I have 2 different types of data streams, one providing weather data and the other providing vehicle data, and I would like to use Flink to do complex event processing on the data.

Which method in Flink 1.3.x is the correct method to use? I saw different methods like Union, Connect, Window Join. Basically I just want to try a simple CEP like this:

IF weather is wet AND vehicle speed > 60 
WITHIN the last 10 seconds
THEN raise alert

Thanks!

Solution

  • In my opinion, there are two ways how you can solve this problem:

    1. Use a common parent type for different types of events and connect two streams via union method before using CEP library.

    2. You can use flink-siddhi package to process the streams using SiddhiCEP what provides the way to describe a pattern (via SiddhiQL) for several data streams in the same time. More information about flink-siddhi here: https://haoch.github.io/flink-siddhi/. The source code is available on GitHub: https://github.com/haoch/flink-siddhi. Documentation for SiddhiCEP and SiddhiQL: https://docs.wso2.com/display/CEP420/SiddhiQL+Guide+3.1.

    Hope, this information would be helpful.