Search code examples
loopsansibleconditional-statementsblockansible-2.x

Looping or repeating a group of tasks until success


I currently have a playbook which includes a task file. In that task file, I would like to check for a condition. If the exit code of that condition is not equal to 0, all steps in the task file should be repeated. I have tried a few variations with block and loops but I have not figured out a way to make it do what I described above.

Currently I have something like this:

tasks:
  - name: call task file
    include: task_file.yml

In task_file.yml,

- name: perform an operations
  shell: do A
    
- name: check
  shell: do B
  register: result

Next, I would like to tell the main playbook that if result.rc != 0, please repeat task_file.yml until result.rc == 0.

Any pointers would be greatly appreciated

The playbook seems to end no matter what the exit code.


Solution

  • There's no direct way to reach your goal as include_tasks does not support the retry/until loop keywords.

    There is an attempt to circumvent that limitation by teaching ansible a new loop_control.until keyword for loops which could be used for includes. Unfortunately, the pull request has been opened since Sep. 2019 and has still not reached a realease.

    The good news is you can implement that with some work by using include recursion with a block. The below example is largely inspired by a blog article on https://dev.to. I adapted to the current context, fixed some good practice and added features like flexible retries number and delay between retries. Here we go:

    The tasks to be retried go in task_file.yml

    ---
    - name: group of tasks to repeat until success
      block:
        - name: increment attempts counter
          ansible.builtin.set_fact:
            attempt_number: "{{ attempt_number | d(0) | int + 1 }}"
            
        - name: dummy task
          ansible.builtin.debug:
            msg: "I'm a dummy task"
    
        - name: task to check for success.
          # Just for the example. Will return success on attempt number 3
          ansible.builtin.command: "[ {{ attempt_number | int }} -eq 3 ]"
          changed_when: false
    
      rescue:
        - name: "Fail if we reached the max of {{ max_attempts | d(3) }} attempts"
          # Default will be 3 attempts if max_attempts is not passed as a parameter
          ansible.builtin.fail:
            msg: Maximum number of attempts reached
          when: attempt_number | int == max_attempts | int | d(3)
    
        - ansible.builtin.debug:
            msg: "group of tasks failed on attempt {{ attempt_number }}. Retrying"
    
        - name: add delay if needed
          # no delay if retry_delay is not passed as parameter
          ansible.builtin.wait_for:
            timeout: "{{ retry_delay | int | d(omit) }}"
          when: retry_delay is defined
    
        # include ourselves to retry.
        - ansible.builtin.include_tasks: task_file.yml
    

    As you can see, the file includes itself again in case of failure until success of max attempts is reached. Also, note that a retry will happen if any task fails inside the block, not only the last one. If you have a more complex scenario, you can implement more checks for fail/not fail in the rescue section an even add an always section if needed. See anbile blocks

    Then you can call this file from your playbook:

    ---
    - hosts: localhost
      gather_facts: false
    
      tasks:
        - name: Include tasks to retry. 7 attempts max with 1 second delay
          ansible.builtin.include_tasks: task_file.yml
          vars:
            max_attempts: 7
            retry_delay: 1
    

    Playing this example succeeds on third attempt as hardcoded and expected. (You can play around with the parameters to test a fail scenario)

    $ ansible-playbook playbook.yml
    
    PLAY [localhost] ***********************************************************************************************************************************************************************************************************************
    
    TASK [Include tasks to retry] **********************************************************************************************************************************************************************************************************
    included: /tmp/toto/task_file.yml for localhost
    
    TASK [increment attempts counter] ******************************************************************************************************************************************************************************************************
    ok: [localhost]
    
    TASK [dummy task] **********************************************************************************************************************************************************************************************************************
    ok: [localhost] => {
        "msg": "I'm a dummy task"
    }
    
    TASK [task to check for success.] *******************************************************************************************************************************************************************************************************
    fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["[", "1", "-eq", "3", "]"], "delta": "0:00:00.002104", "end": "2022-12-08 14:16:27.850578", "msg": "non-zero return code", "rc": 1, "start": "2022-12-08 14:16:27.848474", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
    
    TASK [Fail if we reached the max of 7 attempts] ****************************************************************************************************************************************************************************************
    skipping: [localhost]
    
    TASK [ansible.builtin.debug] ***********************************************************************************************************************************************************************************************************
    ok: [localhost] => {
        "msg": "group of tasks failed on attempt 1. Retrying"
    }
    
    TASK [add delay if needed] *************************************************************************************************************************************************************************************************************
    ok: [localhost]
    
    TASK [ansible.builtin.include_tasks] ***************************************************************************************************************************************************************************************************
    included: /tmp/toto/task_file.yml for localhost
    
    TASK [increment attempts counter] ******************************************************************************************************************************************************************************************************
    ok: [localhost]
    
    TASK [dummy task] **********************************************************************************************************************************************************************************************************************
    ok: [localhost] => {
        "msg": "I'm a dummy task"
    }
    
    TASK [task to check for success.] *******************************************************************************************************************************************************************************************************
    fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["[", "2", "-eq", "3", "]"], "delta": "0:00:00.004009", "end": "2022-12-08 14:16:29.496509", "msg": "non-zero return code", "rc": 1, "start": "2022-12-08 14:16:29.492500", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
    
    TASK [Fail if we reached the max of 7 attempts] ****************************************************************************************************************************************************************************************
    skipping: [localhost]
    
    TASK [ansible.builtin.debug] ***********************************************************************************************************************************************************************************************************
    ok: [localhost] => {
        "msg": "group of tasks failed on attempt 2. Retrying"
    }
    
    TASK [add delay if needed] *************************************************************************************************************************************************************************************************************
    ok: [localhost]
    
    TASK [ansible.builtin.include_tasks] ***************************************************************************************************************************************************************************************************
    included: /tmp/toto/task_file.yml for localhost
    
    TASK [increment attempts counter] ******************************************************************************************************************************************************************************************************
    ok: [localhost]
    
    TASK [dummy task] **********************************************************************************************************************************************************************************************************************
    ok: [localhost] => {
        "msg": "I'm a dummy task"
    }
    
    TASK [task to check for success.] *******************************************************************************************************************************************************************************************************
    ok: [localhost]
    
    PLAY RECAP *****************************************************************************************************************************************************************************************************************************
    localhost                  : ok=14   changed=0    unreachable=0    failed=0    skipped=2    rescued=2    ignored=0