Search code examples
jsontraversaljq

jq 1.5 - Change existing element or add new if does not exist

Using:

Goals and conditions:

  1. Replace an child object value with another value, at any depth, having parents objects or arrays, for example:
    • if .spec.template.spec.containers[n].env[n].name == "CHANGEME" then
    • .spec.template.spec.containers[n].env[n].value = "xx"
    • where n >=0
  2. If any of the parents of .name do not exist, should be able to add them on the fly instead of exiting with an error
  3. The output JSON should have at least the same elements as the input JSON, no existing elements should be lost
  4. No duplicates are allowed within the elements of an array, but the order must be preserved, so functions like unique cannot be used

Sample input JSON:

The structure is actually imposed, so I have to obey it. An object "path" usually is something like: .spec.template.spec.containers[0].spec.env[1].name. You could also have .containers[1], and so on. This is highly variable, and sometimes some elements could exist or not, depends on a schema definition of that particular JSON.

[
  {
    "kind": "StatefulSet",
    "spec": {
      "serviceName": "cassandra",
      "template": {
        "spec": {
          "containers": [
            {
              "name": "cassandra",
              "env": [
                {
                  "name": "CASSANDRA_SEEDS",
                  "value": "cassandra-0.cassandra.kong.svc.cluster.local"
                },
                {
                  "name": "CHANGEME",
                  "value": "K8"
                }
              ]
            }
          ]
        }
      }
    }
  }
]

Scenarios

  1. Replace an existing value while preserving the input structure, works as expected:
    • jq -r 'map({name:"CHANGEME",value: "xx"} as $v | (.spec.template.spec.containers[].env[] | select(.name==$v.name))|=$v)'

  2. Let's assume I want to do the same, only that .env1 is the parent array of the object {name:"",value:""}. The expected output should be:

    [
  {
    "kind": "StatefulSet",
    "spec": {
      "serviceName": "cassandra",
      "template": {
        "spec": {
          "containers": [
            {
              "name": "cassandra",
              "env": [
                {
                  "name": "CASSANDRA_SEEDS",
                  "value": "cassandra-0.cassandra.kong.svc.cluster.local"
                },
                {
                  "name": "CHANGEME",
                  "value": "K8"
                }
              ],
              "env1": [
                {
                  "name": "CHANGEME",
                  "value": "xx"
                }
              ]
            }
          ]
        }
      }
    }
  }
]
    • For this, I have tried to add an object env1 on the fly:
      • jq -r 'map({name:"CHANGEME",value: "xx"} as $v | (.spec.template.spec.containers[] | if .env1 == null then .+={env1:[$v]} | .env1 else .env1 end | .[] | select(.name==$v.name))|=$v)'
        • works if .env1 exists, else:
        • error: Invalid path expression near attempt to access element "env1" of {"name":"cassandra","env"..
        • same results if using notations like .env//[$v] or .env//=.env[$v]
      • jq -r 'map({name:"CHANGEME",value: "xx"} as $v | (.spec.template.spec.containers[].env1 | .[if length<0 then 0 else length end]) |= $v)'
        • works if .env1 does not exist
        • adds another element if the array .env1 exists, potentially duplicating objects
    • Eventually I have managed to create a working filter:
      • jq -r 'def defarr: if length<=0 then .[0] else .[] end; def defarr(item): if length<=0 then .[0] else foreach .[] as $item ([]; if $item.name == item then $item else empty end; .) end; map({name:"CHANGEME",value: "xx"} as $v | (.spec.template.spec | .containers1 | defarr | .env1 | defarr($v.name) ) |=$v)'
        • this works as expected, however is too long and heavy and have to add the custom functions after each potential array in the object hierarchy

The question

Is there any way to simplify all this, make it a bit more generic to handle any number of parents, arrays or not?

Thank you.

Solution

  • Managed to reach a very good form:

    1. Added the following functions in ~/.jq:

      def arr:
    if length<=0 then .[0] else .[] end;

def arr(f):
    if length<=0 then
        .[0]
    else
        .[]|select(f)
    end//.[length];

def when(COND; ACTION):
    if COND? // null then ACTION else . end;

# Apply f to composite entities recursively, and to atoms
def walk(f):
  . as $in
  | if type == "object" then
      reduce keys_unsorted[] as $key
        ( {}; . + { ($key):  ($in[$key] | walk(f)) } ) | f
  elif type == "array" then map( walk(f) ) | f
  else f
  end;

def updobj(f):
  walk(when(type=="object"; f));

    2. A typical filter will look like this:

      jq -r '{name:"CHANGEME",value: "xx"} as $v |
    map( when(.kind == "StatefulSet";
              .spec.template.spec.containers|arr|.env|arr(.name==$v.name)) |= $v)'

    The result will be that all objects that do not exist already will be created. The convention here is to use the arr functions for each object that you want to be an array, and at the end use a boolean condition and an object to replace the matched one or add to the parent array, if not matched.

    1. If you know the path is always there, and so are the object you want to update, walk is more elegant:

      jq -r 'map(updobj(select(.name=="CHANGEME").value|="xx"))'

    Thank you @peak for your effort and inspiring the solution.