I created a 3 member Replica Set so I can use one of the members for reading operations without interfering with the main database performance.
To achieve this, the main db system is configured as primary, the other is a secondary with priority 0 so as not to take over the primary role in case of an outage and the third is an Arbiter to help break a tie per the requirements of MongoDB of having an odd number of members.
While playing with the setup, I noticed that if I kill the Arbiter and my Secondary member, the Primary becomes a Secondary. This would affect the main database's read and write access since its role has changed. To resolve this, I would have to restart the main db and remove it from the replica set until my secondary and/or arbiter get back online.
While the probability of two systems failing or having networking issues is low, I have introduced this vulnerability to the main database. Is there a way around this to avoid that issue. One thought that crossed my mind is to add yet another arbiter on another machine, however I am not sure if that is feasible and would then make the set even.
When the secondary and arbiter replica are killed, the primary becomes secondary because it doesn't get the majority of votes (only 1/3 in this case which is less than 50%) to become primary. The purpose is to prevent more than one replica becoming primary during a network outage to maintain data consistency.
Adding another arbiter in this case doesn't help because you will have only 2/4 votes which is not more than 50%. This is the reason why we don't need a even number of replica sets. You can add two arbiters to make an odd number in this case.
If you really don't want the secondary becoming primary (e.g, secondary is there for read-only) and prevent primary from becoming a secondary, you can set vote number to 0 for the secondary and arbiter (See http://docs.mongodb.org/manual/reference/replica-configuration/#local.system.replset.members[n].votes). In this case, you can remove the arbiter as it become useless.
cfg = rs.conf()
cfg.members[0].votes = 1 // the primary (by default votes = 1)
cfg.members[1].votes = 0 // the secondary
cfg.members[2].votes = 0 // the arbiter
rs.reconfig(cfg)
Another solution is you can give 3 votes to the primary. The total number of votes is then 5, and the primary always win the election with at least 3/5 votes. But this is deprecated since version 2.6 which does not allow to have votes higher than 1.
cfg = rs.conf()
cfg.members[0].votes = 3 // the primary
cfg.members[1].votes = 1 // the secondary
cfg.members[2].votes = 1 // the arbiter
rs.reconfig(cfg)