Search code examples
amazon-web-servicesdevopsamazon-ecsservice-discovery

Not able to make AWS ECS services communicate over service discovery

I am trying to make 2 services communicate over service discovery endpoint in AWS ECS service.

Example:

Service1: runs the Task Definition to run nginx and phpfpm

Service2: runs the Task Definition to run redis

Now, I need to make service1 container communicate to service2 container

As per the documentations and resource found on internet. This is what I have done and not able to achieve the need.

  1. We need to turn on service discovery (Done)
  2. Set proper service name and namespace which will work as service discovery endpoint (Done)
  3. Create task definition and create service with above property set (done)
  4. Now AWS will generate a SRV records on the Route53 (OK)

Now, when using the service discovery endpoint which is generally in format service_discovery_service_name.service_discovery_namespace.

The error logs shows , It's not able to resolve the name.

enter image description here

Solution

  • Update 03/2022

    AWS has now ENI Trunking which can increase how many ENIs can be attached to a given EC2 Instance Type in the VPC. This makes using awsvpc mode lot flexible with DNS A records and makes Service Discovery easier to configure for ECS Services.

    Combining this with AWS App Mesh and AWS Cloud Map you can make ECS Service Discovery lot easier.

    More info about ENI Trunking & App Mesh Examples. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-instance-eni.html https://github.com/aws/aws-app-mesh-examples/tree/main/walkthroughs/howto-ingress-gateway

    Original Answer

    As per our conversation, here is bit summary of what's happening.

    • If Service1(nginx in your case) needs to interact with Service2(redis) with AWS ServiceDiscovery option and use of SRV records then Service1 needs to be aware that it needs to perform DNS SRV lookup instead of DNS A(Address) lookup.

    • You have multiple options here. First, if you want to continue to use the SRV records use then your client nginx needs to proxy redis upstream server with options of service and resolve which are available only in premium version of nginx. Check my sample nginx configuration I have tested at the bottom of the answer which works.

    • Also make sure, you create the AWS Service discovery name with prefix _http._tcp otherwise, I had issues configuration SRV resolve/service option in nginx configuration without the prefix.

    aws ecs service

    • Other option, If you do not want to rely on SRV records but go to standard A record lookup then you will have to use awsvpc mode for containers and select A option.

    enter image description here

    • With DNS A option then your query of service_discovery_service_name.service_discovery_namespace will work fine.
    • With DNS A option, there are some constraints. You cannot run multiple tasks on a given EC2 instance due to number of ENIs limit which can be attached depending EC2 instance family. Update Check 03/2022 modification above.

    Sample nginx DNS SRV Options configuration:

    stream {
    resolver 172.31.0.2;
    upstream redis {
        zone tcp_servers 64k;
        server redisservice.local service=_http._tcp resolve;
    }
    server {
        listen 12345;
        status_zone tcp_server;
        proxy_pass redis;
    }
}

    Some references -

    https://aws.amazon.com/blogs/aws/amazon-ecs-service-discovery/ https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service-discovery.html