Consul agent. Check socket connection failed: error="dial tcp 172.19.0.6:50044: connect: connection refused"

Question

I am having troubles with microservice health checks in my consul docker setup, which i believe is a symptom of failure in service discovery as i only have one server in my registry.

Below is consul list of members from inside the docker container.

/ # consul members
Node          Address          Status  Type    Build  Protocol  DC   Segment
7b1edb14a647  172.19.0.6:8301  alive   server  1.7.4  2         dc1  <all>
/ #

Consul container logs repeat the same error below for all the microservices:

consul | 2020-06-16T12:19:11.087Z [WARN]  agent: Check socket connection failed: check=service:ffa44b66c4869601c04abdbea6dc5be5 error="dial tcp 172.19.0.6:50044: connect: connection refused"

I am using docker-compose v.3.2 to create a network for containers.

This is a consul service definition

  consul:
    container_name: consul
    ports:
      - '8400:8400'
      - '8500:8500'
      - '8600:53/udp'
    image: consul
    command: ['agent', '-server', '-bootstrap', '-ui', '-client', '0.0.0.0']

Microservice definition

  service-notification:
    build:
      context: .
      dockerfile: apps/service-notification/Dockerfile
      args:
        NODE_ENV: development
    depends_on:
      - consul
    image: 'service-notification:latest'
    restart: always
    environment:
      - CONSUL_HOST=consul
    ports:
      - '50044:50044'

I am using CONSUL_HOST env variable to pass in correct host url.

Consul config for the microservice

  consul:
    host: ${{CONSUL_HOST}}
    port: 8500
  service:
    discoveryHost: ${{CONSUL_HOST}}
    healthCheck:
      timeout: 1s
      interval: 10s
      tcp: ${{ service.discoveryHost }}:${{ service.port }}
    maxRetry: 5
    retryInterval: 5000
    tags: ["v1.0.0", "microservice"]
    name: io.ultimatebackend.srv.notification
    port: 50044

My conclusion so far is that consul server container fails to reach the agents somehow. But i don't know why and i feel like i am missing some obvious peace of consul structure. Please advise.

Answer 1

I was incorrectly configuring my service. The dicoveryHost should be an IP and port of a micro-service inside docker network.