Search code examples
amazon-dynamodbamazon-dynamodb-streams

Using dynamodb streams with a single table design - handle only specific item types

Ive been building a serverless app using Dynamodb as the database, and have been following the single table design pattern (e.g. https://www.alexdebrie.com/posts/dynamodb-single-table/). Something that I'm starting to come up against is the use of dynamodb streams - I want to be able to use a dynamodb stream to keep an Elasticsearch instance up to date.

At the minute the single dynamodb table holds about 10 different types of items (which will continue to expand), one of these item types, 'event' (as in a sporting event) will be sent to the elastic search instance for complex querying/searching. Therefore any changes to an 'event' item will need to be updated in Elasticsearch via a lambda function triggered by the stream.

What I am struggling with is that I will have a lambda being triggered on 'update' on any of the table items, but that could also be an update of one of the other 9+ item types, I get that inside the lambda I can check for the item that was updated and check its type etc, but it seems wasteful that pretty much any update to any item type will trigger the lambda, which could be potentially a lot more times than needed.

Is there a better way to handle this to be less wasteful and more targeted to only one item type? I'm thinking that as the app grows and more stream triggers are needed, at least there would be an 'update' lambda already being triggered that I could run some logic to see what type of item was updated, but I'm just concerned i've missed a point on something.

Solution

  • You can use Lambda Event Filtering. This will allow you to prevent specific events from ever invoking your function. In the case of your single table DynamoDB design pattern, you can filter out only records with type: EVENT.

    If you so happen to be utilizing the Serverless Framework, the following yaml snippet showcases how you can easily implement this feature.

    functionName:
  handler: src/functionName/function.handler
  # other properties
  events:
  - stream:
      type: dynamodb
      arn: !GetAtt DynamoDbTable.StreamArn
      maximumRetryAttempts: 1
      batchSize: 1
      filterPatterns:
        - eventName: [MODIFY]
          dynamodb:
             MyTableName:
               type:
                 S: [EVENT]

    Note multiple comparison operators exist such as begins with i.e. [{"prefix":"EVENT"}] ~ see Filter rule syntax for more.