Is it possible to dynamically build an Elastic cluster with a single run list?

I'm using the Elasticsearch cookbook.

My ultimate goal:

Have 4 or more previously deployed VMs that are unconfigured to run ES
I apply the same recipe/runlist to all of these nodes and it installs and configures an ES cluster for me

Before I start going down this road I want to know if this is even possible or are there node specific configurations that make it impossible to accomplish this with a single run list.

If the nodes are preconfigured with my desired hostnames and IPs I can use node attributes to plug this info in the configs. Can I have a recipe randomly choose 3 of the four or more nodes to become primary shards?

Solution

It is most definitely possible. There may be a better practice out there (if so please share) but I was able to create a dynamic cluster like this using Chef search and tags (although I was writing a cookbook for consul, not ES).

Search Chef for any nodes tagged as the main ES master. If it doesn't find any, converge the current node as a master. If a master exists, converge the current node as a slave.

Your Chef search results will contain all the node information for ES master. Instead of maintaining preconfigured IPs and hostnames, pull it from your search results' node object (allows for scale).

The following snippet assumes you have a master.rb recipe and a slave.rb recipe.

es_servers = search(
 :node, 
 "tags:elasticsearch AND chef_environment:#{node.chef_environment}"
)

if es_servers.length < 1 || tagged?('elasticsearch')

    # Configure this node with as ES master
    # ..  
    include_recipe 'es::master'

    # tag this node as a server
    tag('elasticsearch') unless tagged?('elasticsearch')

else

    # There are already 1 ES servers.
    include_recipe 'es::slave'

end

caveat: * If you add the recipe to 4 nodes and execute chef-client at the same time, they'll all spin up as the master node. For the initial build of the cluster, build your master(s) first and then your slaves.