Search code examples
amazon-web-servicesamazon-redshiftamazon-rdsamazon-vpc

AWS DMS endpoint connection to Redshift not working

I'm currently trying so setup a replication from RDS (MySQL) to Redshift via DMS. The endpoint to RDS is working, but the one to Redshift is not. Here is my setup:

VPC

RDS, DMS, and Redshift are running in the same VPC and share the same subnets.

Roles

I implemented the required roles for DMS (dms-vpc-role, dms-cloudwatch-logs-role) and the specific one for Redshift (dms-access-for-endpoint) according to the AWS documentation.

Security groups

The security group setup is the same as well. Both RDS and Redshift have two security groups (empty one and ingress that references the empty one). Inbound rules of the ingress security group as follows, outbound rules default to allow all outgoing communication.

RDS:

enter image description here

Redshift:

enter image description here

Replication instance setup

The replication instance has both security groups. I figured that the replication instance itself will need a security group on its own, however, I cannot manage to create a working one.

What have I done before writing this post?

I have tested the following scenarios:

  • Using the default VPC security group that allows all inbound and outbound traffic -> both endpoints work
  • Created a security group that with inbound rules on TCP 3306 (RDS) and 5439 (Redshift) with connections to the RDS and Redshift security groups mentioned above -> RDS works, Redshift doesn't

And I'm getting the following error that I interpret as DMS not being capable of calling Redshift of any sorts.

Test Endpoint failed: Application-Status: 1020912, Application-Message: N/A, Application-Detailed-Message: N/A

Overall, the fact that it is working with the default VPC security group leads me to the conclusion that (1) it is a network issue and (2) that a simple change to the replication instance security group will probably do the job. However, after countless iterations, I have no idea what I'm missing here.

All ideas and suggestions are highly appreciated!

Solution

  • A typical Security Group configuration for your scenario should be:

    • A security group on the AWS DMS instance (DMS-SG) that permits All Outbound (which is the normal default), plus whatever inbound you need to use DMS
    • A security group on the Amazon RDS instance (RDS-SG) that permits inbound traffic on port 3306 from DMS-SG
    • A security group on the Amazon Redshift instance (Redshift-SG) that permits inbound traffic on port 5439 from DMS-SG

    That is, the DMS instance should have its own security group that can be referenced from the other security groups. Security Groups apply to each instance individually. Instances that have the 'same' security group do not gain any special ability to communicate with each other unless it is specifically added as a rule to the security group.