MongoDB docker replica set connection error "Host not found"

Question

I have a local MongoDB replica set created following this SO answer.

The docker-compose file:

services:
  mongo1:
    container_name: mongo1
    image: mongo:4.2
    ports:
      - 27017:27017
    restart: always
    command: ["--bind_ip_all", "--replSet", "rs" ]
  mongo2:
    container_name: mongo2
    image: mongo:4.2
    ports:
      - 27018:27017
    restart: always
    command: ["--bind_ip_all", "--replSet", "rs" ]
  mongo3:
    container_name: mongo3
    image: mongo:4.2
    ports:
      - 27019:27017
    restart: always
    command: ["--bind_ip_all", "--replSet", "rs" ]
  replica_set:
    image: mongo:4.2
    container_name: replica_set
    depends_on:
      - mongo1
      - mongo2
      - mongo3
    volume:
      - ./initiate_replica_set.sh:/initiate_replica_set.sh
    entrypoint: 
      - /initiate_replica_set.sh

The initiate_replica_set.sh file:

#!/bin/bash

echo "Starting replica set initialize"
until mongo --host mongo1 --eval "print(\"waited for connection\")"
do
    sleep 2
done
echo "Connection finished"
echo "Creating replica set"
mongo --host mongo1 <<EOF
rs.initiate(
  {
    _id : 'rs0',
    members: [
      { _id : 0, host : "mongo1:27017" },
      { _id : 1, host : "mongo2:27017" },
      { _id : 2, host : "mongo3:27017" }
    ]
  }
)
EOF
echo "replica set created"

The replica set is brought up successfully and runs fine, but it errors when I try to connect to the replica set:

$ mongo "mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs"
MongoDB shell version v5.0.2
connecting to: mongodb://localhost:27017,localhost:27018,localhost:27019/?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs
{"t":{"$date":"2021-08-05T21:35:40.667Z"},"s":"I",  "c":"NETWORK",  "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"rs","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set rs"}}
Error: Could not find host matching read preference { mode: "nearest" } for set rs, rs/localhost:27017,localhost:27018,localhost:27019 :
connect@src/mongo/shell/mongo.js:372:17
@(connect):2:6
exception: connect failed
exiting with code 1

More verbose log:

{
  "t": {
    "$date": "2021-08-05T21:35:54.531Z"
  },
  "s": "I",
  "c": "-",
  "id": 4333222,
  "ctx": "ReplicaSetMonitor-TaskExecutor",
  "msg": "RSM received error response",
  "attr": {
    "host": "mongo1:27017",
    "error": "HostUnreachable: Error connecting to mongo1:27017 :: caused by :: Could not find address for mongo1:27017: SocketException: Host not found (authoritative)",
    "replicaSet": "rs",
    "response": "{}"
  }
}

What is the cause of the problem and how do I fix it?

Answer 1

There are some partial answers on this issue from various places, here is what I think as a complete answer.

The Cause

• Mongo clients use the hostnames listed in the replica set config, not the seed list

  Although the connection string is "mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs", mongo client does not connect to the members of the replica set with seed addresses localhost:27017 etc, instead the client connects to the members in the replica config set returned from the seed hosts, i.e., the ones in the rs.initiate call. This is why the error message is Error connecting to mongo1:27017 instead of Error connecting to localhost:27017.

• Container hostnames are not addressable outside the container network

  A mongo client inside the same container network as the mongo server containers can connect to the server via addresses like mongo1:27017; however, a client on the host, which is outside of the container network, can not resolve mongo1 to an IP. The typical solution for this problem is proxy, see Access docker container from host using containers name for details.

The Fix

Because the problem involves docker networking and docker networking varies between Linux and Mac. The fixes are different on the two platforms.

Linux

The proxy fix (via 3rd party software or modifying /etc/hosts file) works fine but sometimes is not viable, e.g., running on remote CI hosts. A simple self-contained portable solution is to update the intiate_replia_set.sh script to initiate the replica set with member IPs instead of hostnames.

intiate_replia_set.sh

echo "Starting replica set initialization"
until mongo --host mongo1 --eval "print(\"waited for connection\")"
do
   sleep 2
done
echo "Connection finished"
echo "Creating replica set"

MONGO1IP=$(getent hosts mongo1 | awk '{ print $1 }')
MONGO2IP=$(getent hosts mongo2 | awk '{ print $1 }')
MONGO3IP=$(getent hosts mongo3 | awk '{ print $1 }')

read -r -d '' CMD <<EOF
rs.initiate(
  {
    _id : 'rs',
    members: [
      { _id : 0, host : '${MONGO1IP}:27017' },
      { _id : 1, host : '${MONGO2IP}:27017' },
      { _id : 2, host : '${MONGO3IP}:27017' }
    ]
  }
)
EOF

echo $CMD | mongo --host mongo1
echo "replica set created"

This way the mongo replica set members have container IP instead of hostname in their addresses. And the container IP is reachable from the host.

Alternatively, we can assign static IP to each container explicitly in the docker-compose file, and use static IPs when initiating the replica set. It is a similar fix but with more work.

Mac

The above solution unfortunately does not work for Mac, because docker container IP on Mac is not exposed on the host network interface. https://docs.docker.com/docker-for-mac/networking/#per-container-ip-addressing-is-not-possible

The easiest way to make it work is to add following mapping in /etc/hosts file:

127.0.0.1   mongo1 mongo2 mongo3