Ansible - How to extract specific keys from a List of Dictionary by iterating through another List [edit]

Question

I have a list of server names and a List of dicts for all clusters in an environment. The List of dictionaries contains the relevant servers in that cluster. e.g.

"full_cluster_dict": [
{
    "key": "cluster_a", 
    "value": [
        "ca_server1",
        "ca_server2",
        "ca_server3",
        "ca_server4",
        "ca_server5",
        "ca_server6",
        "ca_server7",
        "ca_server8"
    ]
},
{
    "key": "cluster_b", 
    "value": [
        "cb_server1",
        "cb_server2",
        "cb_server3"
    ]
},
{
    "key": "cluster_c",
    "value": [
        "cc_server1",
        "cc_server2",
        "cc_server3",
        "cc_server4"
    ]
}

and

"server_list": [
    "ca_server1",
    "cb_server2",
    "ca_server6"
]

I would like to create a smaller list of dicts showing only clusters that contain servers from server_list . e.g.

"needed_cluster_dict": [
  {
     "key: "cluster_a",
     "value": [
        "ca_server1",
        "ca_server2",
        "ca_server3",
        "ca_server4",
        "ca_server5",
        "ca_server6",
        "ca_server7",
        "ca_server8"
    ]
 },
 {
     "key": "cluster_b",
     "value" : [
        "cb_server1",
        "cb_server2",
        "cb_server3"
    ]
 }
]

I tried the following

- name: extract only relevant clusters based on the list of servers
  ansible.builtin.set_fact:
    needed_cluster_dict: "{{ needed_cluster_dict|d({}) | combine({item: cluster_filter}) }}"
  with_items: "{{ server_list }}"
  vars:
    cluster_filter: "{{ sds_dict|dict2items|json_query(_query) }}"
    _query: '[?value.contains(@, `{{ item }}`)].value'`

but this only returns a dictionary where the server names above are the keys and each key contains a list of servers e.g.

needed_cluster_list: {
  "ca_server1: [
    "ca_server1",
    "ca_server2",
    "ca_server3",
    "ca_server4"
  ],
  "ca_server2: [
    "ca_server1",
    "ca_server2",
    "ca_server3",
    "ca_server4"
  ],
  ...
  ...
}

Apologies for the edit from my original question but after asking, i discovered it's easier to loop through a list of Dictionaries than it is to loop through a Dictionary.

Answer 1

Update

Create list intersect

  intersect: "{{ full_cluster_dict|
                 map(attribute='value')|
                 map('intersect', server_list)|
                 map('community.general.dict_kv', 'intersect') }

gives

 intersect:
    - intersect: [ca_server1, ca_server6]
    - intersect: [cb_server2]
    - intersect: []

Combine the lists' items and select items by non-empty intersect

  needed_cluster_dict: "{{ full_cluster_dict|zip(intersect)|
                           map('combine')|
                           selectattr('intersect') }}"

gives

  needed_cluster_dict:
    - intersect: [ca_server1, ca_server6]
      key: cluster_a
      value: [ca_server1, ca_server2, ca_server3, ca_server4, ca_server5, ca_server6,
        ca_server7, ca_server8]
    - intersect: [cb_server2]
      key: cluster_b
      value: [cb_server1, cb_server2, cb_server3]

You can remove the attribute intersect

  needed_cluster_dict: "{{ full_cluster_dict|zip(intersect)|
                           map('combine')|
                           selectattr('intersect')|
                           ansible.utils.remove_keys(target=['intersect'])}}"

gives

  needed_cluster_dict:
    - key: cluster_a
      value: [ca_server1, ca_server2, ca_server3, ca_server4, ca_server5, ca_server6,
        ca_server7, ca_server8]
    - key: cluster_b
      value: [cb_server1, cb_server2, cb_server3]

---

Example of a complete playbook for testing

- hosts: localhost

  vars:

    full_cluster_dict:
    - key: cluster_a
      value:
      - ca_server1
      - ca_server2
      - ca_server3
      - ca_server4
      - ca_server5
      - ca_server6
      - ca_server7
      - ca_server8
    - key: cluster_b
      value:
      - cb_server1
      - cb_server2
      - cb_server3
    - key: cluster_c
      value:
      - cc_server1
      - cc_server2
      - cc_server3
      - cc_server4

    server_list: [ca_server1, cb_server2, ca_server6]

    intersect: "{{ full_cluster_dict|
                   map(attribute='value')|
                   map('intersect', server_list)|
                   map('community.general.dict_kv', 'intersect') }}"
    needed_cluster_dict: "{{ full_cluster_dict|zip(intersect)|
                             map('combine')|
                             selectattr('intersect')|
                             ansible.utils.remove_keys(target=['intersect']) }}"

  tasks:

    - debug:
        var: full_cluster_dict|to_yaml
    - debug:
        var: intersect|to_yaml
    - debug:
        var: needed_cluster_dict|to_yaml

---

Origin

Given the list full_cluster_list (erroneously marked as dictionary full_cluster_dict in the question)

  full_cluster_list:
    - key: cluster_a
      value:
      - ca_server1
      - ca_server2
      - ca_server3
      - ca_server4
      - ca_server5
      - ca_server6
      - ca_server7
      - ca_server8
    - key: cluster_b
      value:
      - cb_server1
      - cb_server2
      - cb_server3
    - key: cluster_c
      value:
      - cc_server1
      - cc_server2
      - cc_server3
      - cc_server4

Convert the list to a dictionary

  full_cluster_dict: "{{ full_cluster_list|items2dict }}"

gives the dictionary used in the answer below

  full_cluster_dict:
    cluster_a:
    - ca_server1
    - ca_server2
    - ca_server3
    - ca_server4
    - ca_server5
    - ca_server6
    - ca_server7
    - ca_server8
    cluster_b:
    - cb_server1
    - cb_server2
    - cb_server3
    cluster_c:
    - cc_server1
    - cc_server2
    - cc_server3
    - cc_server4

---

Q: "Extract specific keys from a list of dictionaries."

A: Create a list of needed clusters. Test the intersection of the lists

  needed_cluster_str: |
    [{% for k,v in full_cluster_dict.items() %}
    {% if v|intersect(server_list)|length > 0 %}
    {{ k }},
    {% endif %}
    {% endfor %}]
  needed_cluster: "{{ needed_cluster_str|from_yaml }}"

gives

  needed_cluster:
  - cluster_a
  - cluster_b

Extract the needed lists

  needed_cluster_lists: "{{ needed_cluster|map('extract', full_cluster_dict)|list }}"

gives

  needed_cluster_lists:
  - - ca_server1
    - ca_server2
    - ca_server3
    - ca_server4
    - ca_server5
    - ca_server6
    - ca_server7
    - ca_server8
  - - cb_server1
    - cb_server2
    - cb_server3

Create the dictionary

  needed_cluster_dict: "{{ dict(needed_cluster|zip(needed_cluster_lists)) }}"

gives

  needed_cluster_dict:
    cluster_a:
    - ca_server1
    - ca_server2
    - ca_server3
    - ca_server4
    - ca_server5
    - ca_server6
    - ca_server7
    - ca_server8
    cluster_b:
    - cb_server1
    - cb_server2
    - cb_server3

---

Example of a complete playbook for testing

- hosts: localhost

  vars:

    server_list: [ca_server1, cb_server2, ca_server6]
    full_cluster_dict:
      cluster_a:
        - ca_server1
        - ca_server2
        - ca_server3
        - ca_server4
        - ca_server5
        - ca_server6
        - ca_server7
        - ca_server8
      cluster_b:
        - cb_server1
        - cb_server2
        - cb_server3
      cluster_c:
        - cc_server1
        - cc_server2
        - cc_server3
        - cc_server4

    needed_cluster_str: |
      [{% for k,v in full_cluster_dict.items() %}
      {% if v|intersect(server_list)|length > 0 %}
      {{ k }},
      {% endif %}
      {% endfor %}]
    needed_cluster: "{{ needed_cluster_str|from_yaml }}"
    needed_cluster_lists: "{{ needed_cluster|map('extract', full_cluster_dict)|list }}"
    needed_cluster_dict: "{{ dict(needed_cluster|zip(needed_cluster_lists)) }}"

  tasks:

    - debug:
        var: needed_cluster
    - debug:
        var: needed_cluster_lists
    - debug:
        var: needed_cluster_dict