search & replace a list of dictionaries by another list of dictionaries and a given attribute

is it possible to search and replace the following (source) list of dictionaries:

- k: aaa
  v: 1
- k: bbb
  v: 2
- k: bbb
  v: 3
  v2: should be kept from source
- k: bbb
- k: ccc
  v: 4

by this (search & replace) list of dictionaries, using 'k' as index:

- k: bbb     # search for index k:bbb in the source and replace/add key-value v:99 
  v: 99
- k: ccc     # search for index k:ccc in the source and replace/add key-values v:88 and v2:77 
  v: 88
  v2: 77
- k: xxxxx   # search for index k:xxxxx in the source. as not found, just skip
  v: 66

to produce

- k: aaa   # index not found: keep everything as it is in the source
  v: 1
- k: bbb   # index found: replace value of v
  v: 99
- k: bbb   # index found: replace value of v, keep v2 from source
  v: 99
  v2: should be kept from source
- k: bbb   # index found: add missing key-value v:99
  v: 99
- k: ccc   # index found: replace value of v, add key-value v2:77
  v: 88
  v2: 77

it is similar to community.general.lists_mergeby, but not the same.

hope, this is possible with some existing filters.

Solution

Given the lists

    l1:
      - {k: aaa, v: 1}
      - {k: bbb, v: 2}
      - {k: bbb, v: 3, v2: should be kept from source}
      - {k: bbb}
      - {k: ccc, v: 4}
    l2:
      - {k: bbb, v: 99}
      - {k: ccc, v: 88, v2: 77}
      - {k: xxxxx, v: 66}

Convert the second list to a dictionary

  d2: "{{ dict(l2|map(attribute='k')|
               zip(l2|ansible.utils.remove_keys(target=['k']))) }}"

gives

  d2:
    bbb: {v: 99}
    ccc: {v: 88, v2: 77}
    xxxxx: {v: 66}

^{Notes:

It is not necessary to remove the key k. The resulting list l3 will be the same.
d2: "{{ dict(l2|map(attribute='k')|zip(l2)) }}"

Optionally, use json_query
d2: "{{ dict(l2|json_query('[].[k, @]')) }}"

gives the same result
d2:
bbb: {k: bbb, v: 99}
ccc: {k: ccc, v: 88, v2: 77}
xxxxx: {k: xxxxx, v: 66}}

Iterate the first list and combine items

  l3: |
    [{% for i in l1 %}
    {{ i|combine(d2[i.k]|d({})) }},
    {% endfor %}]

gives what you want

  l3:
    - {k: aaa, v: 1}
    - {k: bbb, v: 99}
    - {k: bbb, v: 99, v2: should be kept from source}
    - {k: bbb, v: 99}
    - {k: ccc, v: 88, v2: 77}

^{Example of a complete playbook for testing
- hosts: all

vars:

l1:
- {k: aaa, v: 1}
- {k: bbb, v: 2}
- {k: bbb, v: 3, v2: should be kept from source}
- {k: bbb}
- {k: ccc, v: 4}
l2:
- {k: bbb, v: 99}
- {k: ccc, v: 88, v2: 77}
- {k: xxxxx, v: 66}

d2: "{{ dict(l2|map(attribute='k')|zip(l2)) }}"
l3: |
[{% for i in l1 %}
{{ i|combine(d2[i.k]|d({})) }},
{% endfor %}]

tasks:

- debug:
var: d2|to_yaml
- debug:
var: l3|to_yaml}