Iterate through nested dicts, retaining key names, in Ansible

Question

I have a data structure that looks like this:

all_vms:
  clusterA:
    group1:
      - vm-01
      - vm-02
    group2:
      - vm-03
      - vm-04
  clusterB:
    group1:
      - vm-05
      - vm-06

The key names are not known beforehand. There may be shared "group" key names between clusters.

I want to loop through that data structure and run a task with each of the arrays, but I need the names of the keys they're contained within. The looping task would look something like:

- task:
     vms: "{{ item.value }}"
     group: "{{ item.key }}"
     cluster: "{{ item.parentkey }}"
  loop: "{{ all_vms | ??? }}"

And that would unroll to:

- task:
    vms:
      - vm-01
      - vm-02
    group: group1
    cluster: clusterA
- task:
    vms:
      - vm-03
      - vm-04
    group: group2
    cluster: clusterA
- task:
    vms:
      - vm-05
      - vm-06
    group: group3
    cluster: clusterB

I cannot change the main cluster/group structure, but I can change the structure of the elements that are currently arrays. I have considered just duplicating the keys as values, like this:

all_vms:
  clusterA:
    group1:
      cluster: "clusterA"
      group: "group1"
      vms:
        - vm-01
        - vm-02
    group2:
      cluster: "clusterA"
      group: "group2"
      vms:
        - vm-03
        - vm-04
  clusterB:
    group1:
      cluster: "clusterB"
      group: "group1"
      vms:
        - vm-05
        - vm-06

I would rather not do that, because it's terrible, but I can. But I can't even figure out a way to pop each of those things out into an array. (Edit: Actually, I think figured that out right after posting: all_vms | json_query('*.* | []'). I guess I can go with that if there's not a way to use the tidier data structure.)

Or if I could just use a @!#$% nested loop, if ansible would let me:

- block:
  - task:
      vms: "{{ item.value }}"
      group: "{{ item.key }}"
      cluster: "{{ cluster.key }}"
    loop: "{{ cluster.value | dict2items }}"
  loop: "{{ all_vms | dict2items }}"
  loop_control:
    loop_var: cluster

(Yes, I could do this with include_tasks, but having to have a separate file for a nested loop is just ridiculous.)

Any ideas how to iterate over this data structure without having to resort to a separate file just to do nested looping?

Answer 1

And here is the solution using several combinations of filters directly in Ansible / Jinja.

It combines the first level keys and values with a zip filter, in order to have a know subelements name — 1 — on which we can then use a subelements.

The second level key / value pair is accessible thanks to a dict2items mapped on the first level values.

The task ends up being

- set_fact:
    tasks: "{{ tasks | default([]) + [_task] }}"
  loop: >-
    {{
      all_vms.keys()
      | zip(all_vms.values() | map('dict2items'))
      | subelements([1])
    }}
  loop_control:
    label: "{{ item.0.0 }} — {{ item.1.key }}"
  vars:
    _task:
      task:
        vms: "{{ item.1.value }}"
        group: "{{ item.1.key }}"
        cluster: "{{ item.0.0 }}"

---

Given the playbook:

- hosts: localhost
  gather_facts: no

  tasks:
    - set_fact:
        tasks: "{{ tasks | default([]) + [_task] }}"
      loop: >-
        {{
          all_vms.keys()
          | zip(all_vms.values() | map('dict2items'))
          | subelements([1])
        }}
      loop_control:
        label: "{{ item.0.0 }} — {{ item.1.key }}"
      vars:
        _task:
          task:
            vms: "{{ item.1.value }}"
            group: "{{ item.1.key }}"
            cluster: "{{ item.0.0 }}"
        all_vms:
          clusterA:
            group1:
              - vm-01
              - vm-02
            group2:
              - vm-03
              - vm-04
          clusterB:
            group1:
              - vm-05
              - vm-06

    - debug:
        var: tasks

This yields:

TASK [set_fact] ***************************************************************
ok: [localhost] => (item=clusterA — group1)
ok: [localhost] => (item=clusterA — group2)
ok: [localhost] => (item=clusterB — group1)

TASK [debug] ******************************************************************
ok: [localhost] => 
  tasks:
  - task:
      cluster: clusterA
      group: group1
      vms:
      - vm-01
      - vm-02
  - task:
      cluster: clusterA
      group: group2
      vms:
      - vm-03
      - vm-04
  - task:
      cluster: clusterB
      group: group1
      vms:
      - vm-05
      - vm-06