Search code examples
amazon-web-servicesamazon-s3amazon-dynamodbttlamazon-kinesis-firehose

DynamoDB table having TTL and triggers Kinesis Firehose


In my DynamoDB table, Kinesis Firehose is triggered which dumps my data to S3 whenever some records is added/updated. My DynamoDB table also has TTL enabled.

Will it also be triggered when some record is deleted?

When the item expires, will Kinesis Firehose be triggered at that time and what happen on the S3 side?


Solution

  • My understanding is, that the data format DynamoDB sends to Kinesis Data Streams is basically identical as the data it sends to the regular DynamoDB streams and as a result of that, I expect the behavior to be identical.

    According to the Kinesis Data Streams integration docs (emphasis mine):

    Amazon Kinesis Data Streams for Amazon DynamoDB operates asynchronously, so there is no performance impact on a table if a stream is enabled. Whenever items are created, updated, or deleted in the table, DynamoDB sends a data record to Kinesis. The record contains information about a data modification to a single item in a DynamoDB table. Specifically, a data record contains the primary key attribute of the item that was modified, together with the "before" and "after" images of the modified item.

    That's essentially what a regular DynamoDB stream does as well and concerning TTL-deletes the docs for that say:

    You can back up, or otherwise process, items that are deleted by Time to Live (TTL) by enabling Amazon DynamoDB Streams on the table and processing the streams records of the expired items.

    The streams record contains a user identity field Records[].userIdentity.

    Items that are deleted by the Time to Live process after expiration have the following fields:

    • Records[<index>].userIdentity.type

      "Service"

    • Records[<index>].userIdentity.principalId

      "dynamodb.amazonaws.com"

    tl;dr: Yes, the TTL-deletes should show up in the stream as well and will be handled by Firehose like any regular delete.