Merge Two Yaml Files Deeply in Bash yq

Question

I have two yaml files:

file1.yaml

name: "name-a"
namespace: "name-a-space"

livenessProbe:
  path: is-alive

readinessProbe:
  path: is-ready

env:
  - name: key1
    value: test1
  - name: key2
    value: test2
  - name: KEY3
    value: test2

and file2.yaml

name: "name-b"
namespace: "name-b-space"

livenessProbe:
  path: is-alive1

readinessProbe:
  path: is-ready1

env:
  - name: key1
    value: test7

And to merge file1.yaml and file2.yaml I'm using yq v4.30.8

yq '. *= load("file2.yml")' file1.yml based on this reference

The problem is that regarding env key it overwriting all elements of env in file2 on file1, but I need to to just overwrite the different value of key1 so the output should be

name: "name-b"
namespace: "name-b-space"

livenessProbe:
  path: is-alive1

readinessProbe:
  path: is-ready1

env:
  - name: key1
    value: test7
  - name: key2
    value: test2
  - name: KEY3
    value: test2

But my code generate

name: "name-b"
namespace: "name-b-space"

livenessProbe:
  path: is-alive1

readinessProbe:
  path: is-ready1

env:
  - name: key1
    value: test7

Answer 1

Use *d to Merge, deeply-merging arrays which merges arrays "like objects, with indices as their key", i.e. "merge the first item in the array and do nothing with the [other ones].":

yq '. *d load("file2.yaml")' file1.yaml

name: "name-b"
namespace: "name-b-space"
livenessProbe:
  path: is-alive1
readinessProbe:
  path: is-ready1
env:
  - name: key1
    value: test7
  - name: key2
    value: test2
  - name: KEY3
    value: test2

---

As it turned out, you want the arrays in .env to be merged by matching a specific field within the item objects. To this end, you could ireduce the .env array into one object using .name as field names (which becomes the "key" when merging) using .[] as $item ireduce ({}; .[$item.name] = $item.value). Do this for .env on both sides, then use regular merging with *. Eventually, map the indexed object back into your original array structure using map({"name": key, "value": .}) (using unnecessary spaces to highlight duplicate code):

yq '
  (                     .env |= (.[] as $item ireduce ({}; .[$item.name] = $item.value))) *
  (load("file2.yaml") | .env |= (.[] as $item ireduce ({}; .[$item.name] = $item.value)))
  |                     .env |= map({"name": key, "value": .})
' file1.yaml

Also, take a look at how to Merge arrays of objects together, matching on a key in the manual, which is based on the question How to Add / Replace array elements with "yq" conditionally.

Finally, also consider using kislyuk/yq (which is "the other" implementation of yq), or itchyny/gojq. Both are YAML processors with the former internally translating to JSON, then using stedolan/jq under the hood, and the latter being a reimplementation of jq in Go, implementing native YAML functionality. Thus, both can make use of jq features not (yet) implemted by mikefarah/yq. For example:

# using kislyuk/yq
yq -sy '
  map(.env |= INDEX(.name)) | .[0] * .[1] | .env |= map(.)
' file1.yaml file2.yaml

# using itchyny/gojq
gojq -s --yaml-input --yaml-output '
  map(.env |= INDEX(.name)) | .[0] * .[1] | .env |= map(.)
' file1.yaml file2.yaml