Search code examples
jsonexport-to-csvprecisionjq

How do I extract a JSON list into a CSV in command line using JQ?


I have the following JSON data:

% cat test2 

{"day":"2020-07-15","map":
{"a":"ask","b":"bid","t":"timestamp"},"msLatency":52,"pair":"EUR/USD","status":"success","ticks":[
{"b":1.14105,"a":1.14106,"x":48,"t":1594771200000},
{"b":1.14105,"a":1.14106,"x":48,"t":1594771201000},
{"b":1.14103,"a":1.14104,"x":48,"t":1594771202000},
{"b":1.141,"a":1.1413,"x":48,"t":1594771203000},
{"b":1.14103,"a":1.14104,"x":48,"t":1594771205000},
{"b":1.14094,"a":1.14095,"x":48,"t":1594778803000}],"type":"forex"}

And I want to get:

1.14105,1.14106,1594771200000
1.14105,1.14106,1594771201000
1.14103,1.14104,1594771202000
1.141,1.1413,1594771203000
1.14103,1.14104,1594771205000
1.14094,1.14095,1594778803000

Ideally the output shall also pad with zeros, with 1 and 5 being parameters to specify that the first two columns are numbers with 1 natural place and 5 decimal places (although this step can be done easily with awk):

1.14105,1.14106,1594771200000
1.14105,1.14106,1594771201000
1.14103,1.14104,1594771202000
1.14100,1.14130,1594771203000
1.14103,1.14104,1594771205000
1.14094,1.14095,1594778803000

I have tried this with JQ:

 % cat test2 | jq '.ticks'
[
  {
    "b": 1.14105,
    "a": 1.14106,
    "x": 48,
    "t": 1594771200000
  },
  {
    "b": 1.14105,
    "a": 1.14106,
    "x": 48,
    "t": 1594771201000
  },
  {
    "b": 1.14103,
    "a": 1.14104,
    "x": 48,
    "t": 1594771202000
  },
  {
    "b": 1.141,
    "a": 1.1413,
    "x": 48,
    "t": 1594771203000
  },
  {
    "b": 1.14103,
    "a": 1.14104,
    "x": 48,
    "t": 1594771205000
  },
  {
    "b": 1.14094,
    "a": 1.14095,
    "x": 48,
    "t": 1594778803000
  }
]

But I am stuck on how to turn this into a CSV.

EDIT: Just as reference, I previously had the following parsing, using JQ here is much simpler option:

cat test2 |
sed -e 's/{\"/#{\"/g' |
tr '#' '\n' |
grep -v "timestamp" |
grep -v "day" |
sed '/^[[:space:]]*$/d' |
sed 's/].*$//g' |
sed 's/{//g' |
sed 's/},//g' |
sed 's/}//g' |
sed 's/\"//g' |
awk -F '[:,]' -v decimal_places=5 -v integer_places=1 '{
        for(i=1; i<=NF; i=i+2) {
                value[$i]=$(i+1);
        };
        format_price="%0" integer_places "." decimal_places "f"
        format=format_price " " format_price " %d\n"
        printf(format,value["b"],value["a"],value["t"]);
}'

Solution

  • Here's a solution:

    jq -r '.ticks[] | [.b, .a, .t] | join(",")' test2
    
    1.14105,1.14106,1594771200000
    1.14105,1.14106,1594771201000
    1.14103,1.14104,1594771202000
    1.141,1.1413,1594771203000
    1.14103,1.14104,1594771205000
    1.14094,1.14095,1594778803000
    

    It doesn't do the padding to 5 digits. I have no idea how to do that in jq. I still think another language would be easier.