Search code examples
hadoopstream-processingapache-apex

Out of order processing in Apache Apex


The apache-apex documentation states that the order of events in processors is preserved (based on how they are emitted), but does that mean that there is no event-time concept that allows out-of-order processing of events?

Moreover, can events be re-prioritized based on their contents, e.g. when they contain special phrases (e.g. AUTH in a security context). NiFi allows this, which is useful in situations with limited bandwidth.


Solution

  • Apache Apex doesn't support event time processing natively. Event time can be handled on the operator level with time bucketing (which allows for late / out of order arrival).

    It is, however, a roadmap item. You can express your support for it with the community so that it will take a higher priority.

    Native support for event-time windowing https://issues.apache.org/jira/browse/APEXCORE-414

    Overall roadmap http://apex.incubator.apache.org/roadmap.html

    For the second scenario, you can create a separate path in the DAG based on the content.