Search code examples
yamlyq

Merging two yaml documents while concatenating arrays

I want to merge two yaml documents with the result containing

  • all mapped values (with the last one taking precedence)
  • concatenated arrays

e.g. given this file

# file1.yml
animals:
  - elephant
    donkey
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
strange: true

and this file

#file2.yml
animals:
  - monkey
    pelican
flavours:
  sweet:
    - vanilla
strange: false

the result should contain all the nodes, with merged arrays and values that are not arrays from the last file

#result.yml
animals:
  - elephant
    donkey
    monkey
    pelican
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
      vanilla
strange: false

Can yq do this, maybe?

Solution

  • Merging YAML files with yq, concatenating arrays.

    Assumption

    In both file1.yml and file2.yml, the animals array contains a single index with a multiline string.

    elephant
donkey

    for file1.yml, and

    monkey
pelican

    for file2.yml.

    Since you asked about concatenated array, I am assuming file1.yml, file2.yml and results.yml should be like this:

    # file1.yml
animals:
  - elephant
  - donkey
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
strange: true

    # file2.yml
animals:
  - monkey
  - pelican
flavours:
  sweet:
    - vanilla
strange: false

    # result.yml
animals:
  - elephant
  - donkey
  - monkey
  - pelican
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
    - vanilla
strange: false

    Example

    With yq 4.x, you can use the ireduce operator to merge two or more files:

    https://mikefarah.gitbook.io/yq/operators/reduce#merge-all-yaml-files-together

    $ yq4 eval-all '. as $item ireduce ({}; . * $item)' file1.yml file2.yml
animals:
  - monkey
  - pelican
flavours:
  sour:
    - lemon
  sweet:
    - vanilla
strange: false

    Both files have been merged, but duplicated keys and arrays have been overwritten by the latest file.

    To appends arrays instead of overriding them, simply add a + after the merge operator (*).

    https://mikefarah.gitbook.io/yq/operators/multiply-merge#merge-appending-arrays

    $ yq4 eval-all '. as $item ireduce ({}; . *+ $item)' file1.yml file2.yml
animals:
  - elephant
  - donkey
  - monkey
  - pelican
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
    - vanilla
strange: false