Search code examples
salt-project

How to implement a state that wait for other minions finishing certain jobs then execute certain state?


How to implement a state that wait for other minions finishing certain jobs then execute certain state?

For example, I have a cluster of minions called minion-aha1 to minion-aha3, and I install hadoop and hbase on these 3 minions. Now, I would like to convert them to HA mode. Suppose minion-aha1 is the leader. So the logic flow would be:

Start hadoop and hbase on all 3 minions
-> minion-aha1 wait till rest of minions are hadoop and hbase are on and healthy
-> minion-aha1 call join (e.g. stop namenode, hdfs namenode -initializeSharedEdits, start namenode)
-> rest minions call nn2 (e.g. hdfs namenode -bootstrapStandby, start namenode)

I already knew how to convert hbase to HA mode, and I could set the leader in grain, just curious on how to shrink above procedure to single-line, i.e.

salt 'minion-aha*' state.apply hadoop.hbase_to_ha 

Or even salt.orch state would be acceptable. The above would fail due to minion-aha1 never know the state of rest of minions. In other words, it might run successfully once if the developer is lucky, but I look for the solution would run successfully every time.

Thank you.


Solution

  • Basically I benefit from this answer with some modification:

    Trigger event on Master and wait for "response event" on Salt Minion

    For custom event sent to the salt master, e.g. mycompany/hbase/status/*/start, I have to send event -> saltutil.sync_all, then the wait_for_event.