how to merge folder-organized .json file into a single JSON file with folder/file names as keys

Question

I have a folder tree storing various .json files converted from a dataset. something like

/(top folder)
 |-- folder1/
 |    |- file1.json {"a":1,"b":[1,2]}
 |    |- file2.json {"a":12,"c":[]}
 |-- folder2/
      |- file1.json {"ta":1,"tb":[1,2]}
      |- file2.json {"ta":12,"tc":1}
      |- folder21/
          |- file3.json {"test":true}

I hope to merge this folder as

{
   "folder1": {
      "file1.json": {"a":1,"b":[1,2]},
      "file2.json": {"a":12,"c":[]}
   },
   "folder2": {
      "file1.json": {"ta":1,"tb":[1,2]},
      "file2.json": {"ta":12,"tc":1},
      "folder21" : {
        "file3.json": {"test":true}
      }
   }
}

I know this is possible to do with python/perl by writing explicit loops by 1) parsing the json file, 2) merge parsed data with the native data struct/objects using file/folder names as keys, and 3) save the merged data structures into a single JSON file. I've implemented one for Octave/MATLAB, and it works, but it is kind of slow.

I am wondering if it is possible to do this efficiently using jq.

I saw merging json files within the same folder is possible using the reduce expression, see https://stackoverflow.com/a/59769797/4271392

I am wondering if jq allows to scan and iterate over sub-folders and merge through each folder.

If this is possible, I'd be appreciated if someone can point me to the related examples or documentation.

Answer 1

As you mention bash in the comments area, you can try this script calling jq :

#!/bin/bash

shopt -s globstar
jq -n 'reduce inputs as $s ({}; setpath(input_filename|split("/");$s) )' */**/*.json

My test output :

{
  "folder1": {
    "file1.json": {
      "a": 1,
      "b": [
        1,
        2
      ]
    }
  },
  "folder2": {
    "folder21": {
      "file3.json": {
        "test": true
      }
    }
  }
}