Search code examples
apache-flinkamazon-kinesisamazon-kinesis-analytics

How to deploy Kinesis Data Analitics without data loss

We have a complex IoT system in which an AWS Kinesis Flink application collects data, processes it, and forwards new data packages to another system. We collect a lot of small data, such as data from sensors (around 100 per sensor per day), and configuration changes (around one per sensor per week). We receive this data on separate Kinesis Streams, perform some calculations based on both types of data, and send (aggregated) data packages to a new stream (around ten sensor messages plus the last configuration will result in one aggregated info package).

To achieve this, we separate messages based on Sensor IDs in KeyedProcessFunctions and keep Sensor data and Configuration data in ValueStates.

Very rough architecture: enter image description here

What we need to do is start a CI/CD process with this architecture, with frequently updating it as we develop its capabilities and further calculations. The problem is when we update the application, we lose the in-memory state of the application, which could result in losing up to 10 sensor messages per sensor in the worst case, as they are still in the aggregation process and will never be aggregated and forwarded as a new data package.

The even bigger issue is that we lose configuration data as well, resulting in losing a whole day's data per sensor, as the previous Flink app instance received the config, and now it is lost, and we cannot process the data until we receive a new configuration, possibly only by tomorrow.

Is there a good, trustworthy way to solve this problem? To ensure that we keep the ValueStates if we upgrade the app and replay the "only once per day" data stream upon updating the app?

Solution

  • It seems like that snapshots, and restoring snapshots while updating is what I am looging for. From here https://docs.aws.amazon.com/kinesisanalytics/latest/java/how-fault-snapshot.html#how-fault-snapshot-restore :

    "If SnapshotsEnabled is set to true in the ApplicationSnapshotConfiguration for the application, Kinesis Data Analytics automatically creates and uses snapshots when the application is updated, scaled, or stopped to provide exactly-once processing semantics."