Search code examples
oozieoozie-coordinator

How to force coordinator action materialization at specific frequency?


I would like to know if it is possible/how to force a coordinator to materialize or instantiate workflow at regular intervals even if previous instantiated workflow are not done yet.

Let me explain: I have a simple coordinator looking like this:

<coordinator-app name="myApp" frequency="${coord:hours(3)}" start="2015-01-01T0:00Z" end="2016-01-01T00:00Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.4">
   <action>
      <workflow>
         <app-path>${myPath}/workflow.xml</app-path>
      </workflow>
   </action>
</coordinator-app>

The frequency is set to 3 hours. Every 3 hours, I expect the coordinator to "materialize" a new workflow instance/job.

Here is my problem: when the workflow execution lasts more than 3 hours, the coordinator doesn't materialize a new workflow instance but waits for the current running workflow to finish first. It will then instantiate the next workflow. Coordinator-launched workflows are queuing if they last more than the frequency.

How to make the coordinator start a new job every 3 hours no matter what ? Thank you


Solution

  • You should use concurrency property. By default it is one, that's why you have issues with queuing. Set it as big as you find reasonable.

       <coordinator-app name="[NAME]" frequency="[FREQUENCY]" 
                        start="[DATETIME]" end="[DATETIME]" timezone="[TIMEZONE]" 
                        xmlns="uri:oozie:coordinator:0.1">
          <controls>
            <concurrency>[CONCURRENCY]</concurrency>
          </controls>
    

    From docs:

    Concurrency: A coordinator job can specify the concurrency for its coordinator actions, this is, how many coordinator actions are allowed to run concurrently ( RUNNING status) before the coordinator engine starts throttling them.