update config replication of mongodb sharded database access point mongos during run-time

Question

I'm building a sharded MongoDB cluster, which I want to manage automatically. To achieve this I build the normal setup of a sharded db with:

• 3 replications of the config db
• 3 replications of shard "a" (initially only one but by growing the node size the number of shards increase)
• 3 replications of the mongodb access points "mongos"

To connect the config db with the mongos instance I used the following command:

mongos --configdb "cfg/<ip_of_config_server_1>:<port_of_config_instance>,cfg/<ip_of_config_server_2>:<port_of_config_instance>,cfg/<ip_of_config_server_3>:<port_of_config_instance>" --fork --logpath <some_path> --port <some_port>

Now my problem is if one of the config servers is removed from the cluster (or goes down) and another is added, I need to update the mongos instance.

I just don't know how to do this without shutting down the mongos instance and rerunning this command, and thus causing downtime.

Does any of you know an approach or best practices for this situation?

Answer 1

When your client connects to mongos then you can also put multiple mongos hosts (3 in your case) in the connection string, separated by comma. The client runs each command on appropriate mongos instance. So, you can restart the mongos one by one without interrupting the service.

When you have

--configdb "cfg/<ip_of_config_server_1>:27019,cfg/<ip_of_config_server_2>:27019,cfg/<ip_of_config_server_3>:27019"

and one host goes down (let's say ip_of_config_server_1), then the cluster remains fully working as long as config_server_2 or config_server_3 is up.

Usually, when a server goes down, then you don't remove it. You fix the problem and restart it. Or when it physically breaks, then the replaced hardware gets the same hostname/IP, i.e. the --configdb will not change.

Just a note, I would suggest to use a configuration file rather than command line options. In my opinion the configuration file is much easier to handle and to maintain.