[Mongodb][OpsManager]Mongodb secondary instances are not adding to replica set

Question

I have created 3 mongodb instances with below mongod.conf [Primary1,Secondary1,Secondary2]

net:
  port: 27017
  bindIp: 0.0.0.0

replication:
  replSetName: Replica1

I have started the 3 instance using " sudo service mongod start " If I connect primary server to other 2 servers using mongo –host "ip" –port 27017..its working and connected.

Issue 1:

After I do rs.initiate() and rs.conf() Secondary 1 is included in members but when I use rs.status() but showing stateStr" : "STARTUP" And If I check its log means below error :

"msg":"Failed to reap transaction table","attr":{"error":"NotYetInitialized: Replication has not yet been configured"}}

Issue 2:

Because of issue 1 if I try to rs.add() for Secondary 2 ,its not working

rs.status() reponse

{
                        "_id" : 0,
                        "name" : "primary:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY", 
                         .......
}

{
                        "_id" : 1,
                        "name" : "secondary1:27017",
                        "health" : 1,
                        "state" : 0,
                        "stateStr" : "STARTUP",
                        "uptime" : 1579,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2021-01-08T04:47:35.455Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -2,
                        "configTerm" : -1
                }

Log Output of secondary 1's mongodb.log

{"t":{"$date":"2021-01-08T04:39:16.634+00:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"ReplNetwork","msg":"Connecting","attr":{"hostAndPort":"primary:27017"}}
{"t":{"$date":"2021-01-08T04:39:18.004+00:00"},"s":"I",  "c":"CONTROL",  "id":20714,   "ctx":"LogicalSessionCacheRefresh","msg":"Failed to refresh session cache, will try again at the next refresh interval","attr":{"error":"NotYetInitialized: Replication has not yet been configured"}}
{"t":{"$date":"2021-01-08T04:39:18.004+00:00"},"s":"I",  "c":"CONTROL",  "id":20712,   "ctx":"LogicalSessionCacheReap","msg":"Sessions collection is not set up; waiting until next sessions reap interval","attr":{"error":"NamespaceNotFound: config.system.sessions does not exist"}}
{"t":{"$date":"2021-01-08T04:39:36.634+00:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"ReplNetwork","msg":"Connecting","attr":{"hostAndPort":"primary:27017"}}

Answer 1

Issue got fixed. Primary database was able to connect to secondary hosts but in return secondary host can't. So I adjusted the security groups and connected. Takeaway is all the primary and secondary hosts should be able to connect with each other in mongodb port (27017)