Search code examples
jsoncomparisondiffjq

jq - create a new object from the diff in key values of two JSON files

How do a create a JSON object which shows the difference in values of each key if such exist from two different JSON files?

This will be a long post, but mainly due to the JSON files I am using. Please bear with me.

I have a source file - original.json

{
  "billLogAnalyticsAsDavisDataUnits": false,
  "billingProvider": "INTERNAL",
  "blockUIDate": 1652572799000,
  "chatEnabled": false,
  "customMetricsLimit": 9223372036854776000,
  "customMetricsOverageLimit": 9223372036854776000,
  "davisDataUnitsAnnualLimit": -1,
  "davisDataUnitsEnabled": false,
  "davisDataUnitsMigrated": false,
  "davisDataUnitsMonthlyLimit": -1,
  "demUnitsAnnualQuota": 0,
  "demUnitsQuota": 0,
  "expirationCounterEnabled": true,
  "expirationTime": 1652572799000,
  "externalApiQuota": 2147483647,
  "hostUnitsCapping": {
    "fullstackHostLimit": 0,
    "hasContainersHostLimit": 0,
    "hostUnitsCappingEnabled": false,
    "infrastructureOnlyHostLimit": 0
  },
  "hostUnitsQuota": 2147483647,
  "ibmSystemZCICSIMSMonitoring": false,
  "infrastructureSupportedTechnologies": {
    "infrastructureOnlySupport": true,
    "logAgent": true,
    "maxInfrastructureOnlyAgents": 9223372036854776000,
    "networkAgent": true,
    "pluginAgent": true
  },
  "iotEntityQuota": 20,
  "iotTsQuota": 10,
  "isConsumption": false,
  "isCreditExhausted": false,
  "isRumEnabled": false,
  "licenseType": "PAYING",
  "logAnalyticsIngressQuota": 9223372036854776000,
  "logAnalyticsIngressQuotaAnnually": 9223372036854776000,
  "logAnalyticsStorageEnabled": false,
  "logAnalyticsStorageQuota": 0,
  "maxActionsPerMinute": 3500,
  "maxAgents": 2147483647,
  "maxHostUnitsQuota": -1,
  "maxPaasAgents": 2147483647,
  "maxWebChecks": 9223372036854776000,
  "maxWebChecksAnnual": 9223372036854776000,
  "overageCustomMetrics": true,
  "overageEnabled": false,
  "replayStorageDomQuotaInMb": 10000,
  "replayStorageDomRetention": 86400000,
  "retentionCode": 864000000,
  "retentionRum": 864000000,
  "retentionService": 1209600000,
  "retentionWebcheck": 864000000,
  "rumAdditionalUserPropertiesEnabled": true,
  "rumAdditionalUserPropertiesLowerLimit": 20,
  "sessionReplayEnabled": false,
  "sessionStorageQuota": 2147483647,
  "suspensionType": "NONE",
  "symbolicationFileStorageQuota": 1024,
  "syntheticEnabled": false,
  "useHostUnitWeighting": false,
  "visitsAnnualQuota": -1,
  "visitsQuota": -1
}

I need to replace a number of keys having new values which are stored in the template.json file

{
    "billLogAnalyticsAsDavisDataUnits": false,
    "billingProvider": "INTERNAL",
    "blockUIDate": 1652572799000,
    "chatEnabled": false,
    "demUnitsAnnualQuota": -1,
    "demUnitsQuota": -1,
    "isRumEnabled": false,
    "licenseType": "PAYING",
    "logAnalyticsIngressQuota": -1,
    "logAnalyticsIngressQuotaAnnually": -1,
    "logAnalyticsStorageEnabled": false,
    "logAnalyticsStorageQuota": 0,
    "overageCustomMetrics": true,
    "overageEnabled": false,
    "replayStorageDomQuotaInMb": 10000,
    "replayStorageDomRetention": 86400000,
    "retentionCode": 864000000,
    "retentionRum": 864000000,
    "retentionService": 1209600000,
    "retentionWebcheck": 864000000,
    "rumAdditionalUserPropertiesEnabled": true,
    "rumAdditionalUserPropertiesLowerLimit": 20,
    "sessionReplayEnabled": false,
    "sessionStorageQuota": 102400,
    "suspensionType": "NONE",
    "symbolicationFileStorageQuota": 1024,
    "syntheticEnabled": true,
    "useHostUnitWeighting": false,
    "visitsAnnualQuota": 0,
    "visitsQuota": -1
}

I am using jq's following call

jq -n --argfile original.json --argfile template template.json '$original |$original +=$template' >updated.json

to get the new updated.json file with updated values and afterward to submit it to the API server.

API server after processing issues a new file with exactly the same structure. However, some key values might change. Here is the downloaded file from the API server - downloaded.json

{
  "billLogAnalyticsAsDavisDataUnits": false,
  "billingProvider": "INTERNAL",
  "blockUIDate": 1652572799000,
  "chatEnabled": false,
  "customMetricsLimit": 9223372036854776000,
  "customMetricsOverageLimit": 9223372036854776000,
  "davisDataUnitsAnnualLimit": -1,
  "davisDataUnitsEnabled": true,
  "davisDataUnitsMigrated": false,
  "davisDataUnitsMonthlyLimit": -1,
  "demUnitsAnnualQuota": -1,
  "demUnitsQuota": -1,
  "expirationCounterEnabled": true,
  "expirationTime": 0,
  "externalApiQuota": 2147483647,
  "hostUnitsCapping": {
    "fullstackHostLimit": 0,
    "hasContainersHostLimit": 0,
    "hostUnitsCappingEnabled": false,
    "infrastructureOnlyHostLimit": 0
  },
  "hostUnitsQuota": -1,
  "ibmSystemZCICSIMSMonitoring": false,
  "infrastructureSupportedTechnologies": {
    "infrastructureOnlySupport": true,
    "logAgent": true,
    "maxInfrastructureOnlyAgents": 9223372036854776000,
    "networkAgent": true,
    "pluginAgent": true
  },
  "iotEntityQuota": 20,
  "iotTsQuota": 10,
  "isConsumption": false,
  "isCreditExhausted": false,
  "isRumEnabled": true,
  "licenseType": "PAYING",
  "logAnalyticsIngressQuota": -1,
  "logAnalyticsIngressQuotaAnnually": -1,
  "logAnalyticsStorageEnabled": false,
  "logAnalyticsStorageQuota": 0,
  "maxActionsPerMinute": 3500,
  "maxAgents": 2147483647,
  "maxHostUnitsQuota": -1,
  "maxPaasAgents": 2147483647,
  "maxWebChecks": 9223372036854776000,
  "maxWebChecksAnnual": 9223372036854776000,
  "overageCustomMetrics": true,
  "overageEnabled": false,
  "replayStorageDomQuotaInMb": 10000,
  "replayStorageDomRetention": 86400000,
  "retentionCode": 864000000,
  "retentionRum": 864000000,
  "retentionService": 1209600000,
  "retentionWebcheck": 864000000,
  "rumAdditionalUserPropertiesEnabled": true,
  "rumAdditionalUserPropertiesLowerLimit": 20,
  "sessionReplayEnabled": false,
  "sessionStorageQuota": 102400,
  "suspensionType": "NONE",
  "symbolicationFileStorageQuota": 1024,
  "syntheticEnabled": true,
  "useHostUnitWeighting": false,
  "visitsAnnualQuota": 0,
  "visitsQuota": -1
}

Here is my task - I need to find the difference between the updated.json and downloaded.json

I am using diff

diff <(jq -S . update.json) <(jq -S . downloaded.json)

It produces the following result

<   "davisDataUnitsEnabled": false,
---
>   "davisDataUnitsEnabled": true,
15c15
<   "expirationTime": 1652572799000,
---
>   "expirationTime": 0,
23c23
<   "hostUnitsQuota": 2147483647,
---
>   "hostUnitsQuota": -1,
36c36
<   "isRumEnabled": false,
---
>   "isRumEnabled": true,

What I want instead - to create a new object with the original (from updated.json) and new (from downloaded.json) files, so it should look the following way:

    "Original": {
        "davisDataUnitsEnabled": false,
        "expirationTime": 1652572799000,
        "hostUnitsQuota": 2147483647,
        "isRumEnabled": false
    },
    "Updated": {
        "davisDataUnitsEnabled": true,
        "expirationTime": 0,
        "hostUnitsQuota": -1,
        "isRumEnabled": true
    }
}

I tried to use to_entries and from_entries see the difference between the updated and downloaded with the following jq command combining all together:

jq -n \
--argfile original original.json \
--argfile download downloaded.json \
--argfile template template.json \
'$original |$original +=$template | 
($original | to_entries) as $x | 
($download | to_entries) as $y | 
$y - $x | from_entries'

However, the output is quite different compared to the diff:

{
  "davisDataUnitsEnabled": true,
  "demUnitsAnnualQuota": -1,
  "demUnitsQuota": -1,
  "expirationTime": 0,
  "hostUnitsQuota": -1,
  "isRumEnabled": true,
  "logAnalyticsIngressQuota": -1,
  "logAnalyticsIngressQuotaAnnually": -1,
  "sessionStorageQuota": 102400,
  "syntheticEnabled": true,
  "visitsAnnualQuota": 0
}

Seven more keys are listed in the above output, instead of four from diff, and these seven keys have exactly the same values in the updated.json and downloaded.json

My questions - what causes these seven additional keys to appear in the jq output? Can I get with jq a correct output of keys with differences in values and have the output formatted the way I want?

-- P.S. After some digging, figured out that comm is giving me the output I want

comm --nocheck-order -13 <(jq -S . updated.json) <(jq -S . downloaded.json)

Which produces the output:

  "davisDataUnitsEnabled": true,
  "expirationTime": 0,
  "hostUnitsQuota": -1,
  "isRumEnabled": true,

Now if I could put that output under the "Updated" object via JQ while ignoring the trailing comma, that probably would be the end result...

UPDATE I figured out what was the problem and wrote the answer below. Leaving the question as is, because someone might encounter the same situation.

Solution

  • After posting the question, and re-checking the following jq command, I released that I am not keeping the combined original and template files in a variable, so the original, unmodified file is assigned to the var $x.

    jq -n \
--argfile original original.json \
--argfile download downloaded.json \
--argfile template template.json \
'$original |$original +=$template | 
($original | to_entries) as $x | 
($download | to_entries) as $y | 
$y - $x | from_entries

    Instead, the correct jq command which produces the result I wanted is the following:

    jq -n \
--argfile original original.json \
--argfile download downloaded.json \
--argfile template template.json \
'$original |($original +=$template | to_entries) as $x |
($download | to_entries) as $y | 
($y - $x | from_entries) as $new |
($x - $y | from_entries) as $old |
[{"Original":$old,"Updated": $new}]'

    which produces the right output:

    [
  {
    "Original": {
      "davisDataUnitsEnabled": false,
      "expirationTime": 1652572799000,
      "hostUnitsQuota": 2147483647,
      "isRumEnabled": false
    },
    "Updated": {
      "davisDataUnitsEnabled": true,
      "expirationTime": 0,
      "hostUnitsQuota": -1,
      "isRumEnabled": true
    }
  }
]