How can I transfer a database from DynamoDB to Aurora PostgreSQL?

Question

Hello Stack Overflow community,

I'm currently working on a project where I need to migrate data from DynamoDB to Aurora PostgreSQL. I've attempted to connect PostgreSQL to Lambda resolver in the past without success. Now, I'm exploring alternatives and facing challenges.

Here's what I've tried:

1. PostgreSQL with Lambda Resolver: I attempted to connect PostgreSQL to Lambda resolver previously, but encountered issues and couldn't achieve the desired result.
2. AWS Data Pipeline: I tried using AWS Data Pipeline, but it seems to be currently deprecated.
3. AWS Glue: I also experimented with AWS Glue, but haven't obtained a clear solution. I'm looking for guidance on the best approach to transfer DynamoDB data to Aurora PostgreSQL. Specifically:

What tools or methods can I use to effectively fetch data from DynamoDB?

1. Are there any best practices for transferring data between these two databases?
2. How can I address any structural mismatches between DynamoDB and Aurora PostgreSQL?
3. If anyone has experience with similar scenarios or can provide insights, resources, or examples, I would greatly appreciate the assistance.

Thank you.

Answer 1

The short answer here is that AWS Glue is your best option. You can seamlessly connect with both databases and you can run any transform logic you need to ensure your DynamoDB items confirm to your Postgres Schema.

The AWS Glue docs provide examples of how to connect to all databases they support. Structural mismatches or transformations is something that only you can understand, as your data is unique to you.

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html