Search code examples
javaelasticsearchaws-lambdaamazon-kinesisamazon-kinesis-firehose

AWS RDS to AWS ES

Requirement: On insert/update to a specific table create a data stream and push that to elastic search.

Thought AWS RDS(MariaDb) ----- insert/update to say user table ---> AWS Kinesis stream -- using AWS Lambda--> AWS ES

Question How can I create a kinesis stream on insert/update to a specific table? Can I do it through AWS Lambda?

Solution

  • You can do this if you're using Amazon Aurora.

    One of the recent updates in Amazon Aurora is the support to call a Lambda function from a stored procedure.

    So, to implement your requirement, you can create an SQL trigger that calls an SQL stored procedure which then calls a Lambda function that sends that data to Elastic Search.

    Reference: Invoking a Lambda Function from an Amazon Aurora DB Cluster

    Note regarding transactions:

    As @Ahmad Nabil pointed out in the comments below, triggers can be executed before the transaction is committed. So an SQL trigger that calls Lambda may not be what you want.

    If you're not using Aurora but using MySQL or MariaDB, I recommend switching if this feature is really important to your application.

    If you're using a database other than Aurora, MySQL, or MariaDB, you have to rethink your application to write the data in parallel to both RDS and ElasticSearch. I would also recommend the following architecture patterns: