I am currently configuring and adapting some apps to run on the DC/OS. We chose to install the HA-ThreeMaster AWS template.
Recently I was asked to downscale the cluster during development. While removing slave nodes is trivial, I couldn't find any information on how to bring down Master nodes.
I know that they should always have an odd number so I was thinking to remove the two non-leader ec2 instances. Can this be done without destroying the cluster?
It depends on your setup. First check your quorum size: cat /etc/mesos-master/quorum
Than check this http://mesos.apache.org/documentation/latest/operational-guide/
Decreasing the quorum size
The following steps indicate how to decrement the quorum size, using 5 -> 3 masters as an example (quorum size 3 -> 2):
Initially, 5 masters are running with --quorum=3
Remove 2 masters from the cluster, ensure they will not be restarted (see NOTE section above). Now 3 masters are running with --quorum=3
Restart the 3 masters with --quorum=2
To decrease the quorum by N, repeat this process to decrement the quorum size N times.