AWS ECS MongoDB Error: Node primary did not become available

Question

Error:

Node mongodb-primary.ecs.endpoint did not become available

Details:

I have deployed MongoDB on AWS ECS using https://github.com/bitnami/bitnami-docker-mongodb as the Docker image with replica set feature which means that I have 2 more services running on the same EC2 instance apart from primary and those are secondary and arbiter.

Upon deployment, all 3 services start their task and the primary service's task keeps running successfully while secondary and arbiter service's tasks fail.

Here are the logs:

2021-11-28 11:59:58[38;5;6mmongodb [38;5;5m07:59:58.89 [0m[38;5;2mINFO [0m ==> Stopping MongoDB...
2021-11-28 11:59:58[38;5;6mmongodb [38;5;5m07:59:58.89 [0m[38;5;1mERROR[0m ==> Node mongodb-primary.ecs.endpoint did not become available
2021-11-28 11:57:00[38;5;6mmongodb [38;5;5m07:57:00.34 [0m[38;5;2mINFO [0m ==> Found MongoDB server listening at mongodb-primary.ecs.endpoint:27017 !
2021-11-28 11:57:00[38;5;6mmongodb [38;5;5m07:57:00.33 [0m[38;5;2mINFO [0m ==> Trying to connect to MongoDB server mongodb-primary.ecs.endpoint...
2021-11-28 11:56:58[38;5;6mmongodb [38;5;5m07:56:58.20 [0m[38;5;2mINFO [0m ==> Stopping MongoDB...
2021-11-28 11:56:58[38;5;6mmongodb [38;5;5m07:56:58.20 [0m[38;5;2mINFO [0m ==> Configuring MongoDB replica set...

Does anyone know how to fix this?

Answer 1

Issue:

The replica set config was missing.

Solution:

To fix this, I logged into the primary instance using MongoDB Compass and then used mongosh to run the following command: rs.initiate(). Within a minute, other nodes (secondary and arbiter) registered themselves into the replicaset and primary became primary.

Output:

Below are the logs of the rs.status() command:

rs.status()
{
  set: 'rs0',
  date: 2021-11-28T09:34:27.181Z,
  myState: 1,
  term: Long("1"),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long("2000"),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 2,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1638092057, i: 1 }), t: Long("1") },
    lastCommittedWallTime: 2021-11-28T09:34:17.823Z,
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1638092057, i: 1 }), t: Long("1") },
    appliedOpTime: { ts: Timestamp({ t: 1638092057, i: 1 }), t: Long("1") },
    durableOpTime: { ts: Timestamp({ t: 1638092057, i: 1 }), t: Long("1") },
    lastAppliedWallTime: 2021-11-28T09:34:17.823Z,
    lastDurableWallTime: 2021-11-28T09:34:17.823Z
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1638092047, i: 1 }),
  electionCandidateMetrics: {
    lastElectionReason: 'electionTimeout',
    lastElectionDate: 2021-11-28T09:30:17.783Z,
    electionTerm: Long("1"),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 1638091817, i: 1 }), t: Long("-1") },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1638091817, i: 1 }), t: Long("-1") },
    numVotesNeeded: 1,
    priorityAtElection: 1,
    electionTimeoutMillis: Long("10000"),
    newTermStartDate: 2021-11-28T09:30:17.807Z,
    wMajorityWriteAvailabilityDate: 2021-11-28T09:30:17.820Z
  },
  members: [
    {
      _id: 0,
      name: 'ip-x-x-x-x.eu-central-1.compute.internal:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 463,
      optime: [Object],
      optimeDate: 2021-11-28T09:34:17.000Z,
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1638091817, i: 2 }),
      electionDate: 2021-11-28T09:30:17.000Z,
      configVersion: 3,
      configTerm: 1,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: 'mongodb-secondary.ecs.endpoint:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 190,
      optime: [Object],
      optimeDurable: [Object],
      optimeDate: 2021-11-28T09:34:17.000Z,
      optimeDurableDate: 2021-11-28T09:34:17.000Z,
      lastHeartbeat: 2021-11-28T09:34:25.244Z,
      lastHeartbeatRecv: 2021-11-28T09:34:25.292Z,
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: 'ip-x-x-x-x.eu-central-1.compute.internal:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 3,
      configTerm: 1
    },
    {
      _id: 2,
      name: 'mongodb-arbiter.ecs.endpoint:27017',
      health: 1,
      state: 7,
      stateStr: 'ARBITER',
      uptime: 183,
      lastHeartbeat: 2021-11-28T09:34:25.243Z,
      lastHeartbeatRecv: 2021-11-28T09:34:25.388Z,
      pingMs: Long("0"),
      lastHeartbeatMessage: '',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: 3,
      configTerm: 1
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1638092057, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("aaf0912cce77ca5f6013c6a5add706c28d0beed3", "hex"), 0),
      keyId: Long("7035550781860216838")
    }
  },
  operationTime: Timestamp({ t: 1638092057, i: 1 })
}

Conclusion:

After creating the MongoDB cluster on AWS ECS, initiate the replica set with default or with a custom config having all nodes endpoints.