Search code examples
dockerdocker-composeaws-cloudformationamazon-ecs

AWS ECS containers are not connecting but works perfectly in my local machine


I have an application(runs at http://localhost:8080) that talks to a backend api which runs at http://localhost:8081. I have dockerized the frontend and the backend separately and running them through docker-compose locally works perfectly without any issues. But, when I run it in ECS, the frontend couldn't find http://localhost:8081(backend).

I am using an AutoScaling group with an Elastic Load Balancer and I have my both containers defined in a single Task Definition. Also, I have the backend linked to the front end. When I ssh into my ECS instance and run docker ps -a i can see both of my containers are running at the correct ports exactly like in my local machine(Result of docker ps -a) and I can successfully ping each of them from one container to the other.

Task Definition

"CartTaskDefinition": {
   "Type": "AWS::ECS::TaskDefinition",
   "Properties": {
      "ContainerDefinitions": [
       {
          "Name": "cs-cart",
          "Image": "thishandp7/cs-cart",
          "Memory": 400,
          "PortMappings":[
            {
               "ContainerPort": "8080",
               "HostPort": "8080"
            }
          ],
          "Links": [
              "cs-server"
           ]
       },
       {
          "Name": "cs-server",
          "Image": "thishandp7/cs-server",
          "Memory": 450,
          "PortMappings":[
           {
              "ContainerPort": "8081",
              "HostPort": "8081"
           }
          ],
       }
     ]
   }
}

Listeners in my ElasticLoadBalancer, The first listener is for the frontend and the second one is for the backend

"Listeners" : [
  {
    "LoadBalancerPort": 80,
    "InstancePort": 8080,
    "Protocol": "http"
  },
  {
    "LoadBalancerPort": 8081,
    "InstancePort": 8081,
    "Protocol": "tcp"
  }
],

EC2 instacne security Group Ingress rules:

"SecurityGroupIngress" : [
 {
   "IpProtocol" : "tcp",
   "FromPort" : 8080,
   "ToPort" : 8080,
   "SourceSecurityGroupId" : { "Ref": "ElbSecurityGroup" }
 },
 {
   "IpProtocol" : "tcp",
   "FromPort" : 8081,
   "ToPort" : 8081,
   "SourceSecurityGroupId" : { "Ref": "ElbSecurityGroup" }
 },
 {
   "IpProtocol" : "tcp",
   "FromPort" : 22,
   "ToPort" : 22,
   "CidrIp" : "0.0.0.0/0"
 }
],

Docker Compose

version: "3.5"

services:
  cart:
    build:
      context: ..
      dockerfile: docker/Dockerfile
      args:
        APP_LOCATION: /redux-saga-cart/
        PORT: 8080
    networks:
      - server-cart
    ports:
      - 8080:8080
    depends_on:
      - server

  server:
    build:
      context: ..
      dockerfile: docker/Dockerfile
      args:
        APP_LOCATION: /redux-saga-shopping-cart-server/
        PORT: 8081
    ports:
      - 8081:8081
    networks:
      - server-cart


networks:
  server-cart:

Quick update: I have tried it with awsvpc network mode with application load balancer. Still not working

Thanks in advance.


Solution

  • What kind of Docker Network mode are you using(Brdige/Host) on ECS?. I don't think localhost will work properly on ECS containers. I had same issue so I used private IP or DNS name of EC2 host for my communication as temp testing purpose. Ex - http://10.0.1.100:8081.

    Note - Please make sure to give security group rule to allow 8081 traffic from within EC2(Edit EC2 security group to allow 8081 from same sgid as source).

    For Production deployments, I would recommend to use a service discovery to identify the backend service(Consul by Hashicorp) or AWS Private Service Discovery on ECS.


    -- Update --

    Since you are running both containers under same task def(under same ECS service), so typically ECS will bring both docker containers on same host. Do something like following.

    1. By default ECS brings containers using Bridge mode on Linux.
    2. You should be able to have each containers communicate using Docker Gateway IP - 172.17.0.1 on Linux. So for your case, try configuring http://172.17.0.1:8081