Winston JSON Log output to CSV using JQ but getting "Cannot by Csv-Formatted, only array" error

I have a vendor supplied application that outputs Node.JS Winston logs in JSON format that I am trying to output into single line CSV format.

There are many different types of array types and I've used "JQ" utilize to out the elements I want out of each element successfully.

The raw being something like the following:

{"result":{"TransactionID":"ac710dc0-8aa4-11e8-a08c-0d98209bb4f5","TimeStamp":"2018-07-18T12:07:33.082-04:00","SourceSystem":"SOUP","Status":"Accepted"},"level":"info","message":"SOAP createServiceRequest ack recieved & is successful.","timestamp":"2018-07-18T16:07:33.047Z"}

The output being currently with my existing command string (minus the @csv) like:

[
  "2018-07-18T16:07:33.047Z",
  "info",
  "SOAP createServiceRequest ack recieved & is successful.",
  "ac710dc0-8aa4-11e8-a08c-0d98209bb4f5",
  "2018-07-18T12:07:33.082-04:00",
  "Accepted"
]

I have read many posts on SE about outputting to CSV and it seems relatively straight forward but I consistently get an error.

tail -200 ClientOutbound.log | jq-win64.exe --raw-output "[.timestamp, .level, .message, .result .TransactionID, .result .TimeStamp, .result .Status|tostring|@csv]"

Though with the above @csv present I get errors like the following no matter how I narrow down the fields included.. it just moves the error.

jq: error (at <stdin>:199): string ("2018-07-18...) cannot be csv-formatted, only array

and the @csv outside of the array produces a similar error with different detail but also not even parsing the JSON.

 c:\Nodejs\Logs> jq-win64.exe --raw-output @csv ClientOutbound.log "[.level, .message, .result .TransactionID, .result .TimeStamp, .result .Status]"

For example from what type of array:

jq: error (at NCRClientOutbound.log:2074): object ({"level":"i...) cannot be csv-formatted, only array

jq: error (at NCRClientOutbound.log:2075): object ({"argv":["D...) cannot be csv-formatted, only array jq: error (at NCRClientOutbound.log:2076): object ({"CreateSer...) cannot be csv-formatted, only array jq: error (at NCRClientOutbound.log:2077): object ({"lastReque...) cannot be csv-formatted, only array jq: error (at NCRClientOutbound.log:2078): object ({"result":{...) cannot be csv-formatted, only array

The end result I am looking for is something that looks like this (with or without the new encapsulating [] (preferably without) but each row being on the same line.. and as a bonus would be a unique line identifier.

"2018-07-18T16:07:33.047Z",  "info",  "SOAP createServiceRequest ack recieved & is successful.",  "ac710dc0-8aa4-11e8-a08c-0d98209bb4f5",  "2018-07-18T12:07:33.082-04:00",  "Accepted"

What do I need to do to get past my error and get the desired output?

Solution

To add line-numbers, the following helper function would be a good way to go:

def tocsv(s):
  foreach s as $line (0; .+1; [.] + $line)
  | map(tostring) # for robustness
  | @csv ;

And then:

tocsv(inputs
      | [.timestamp, .level, .message]
        + ( .result | [.TransactionID, .TimeStamp, .Status] ))

This assumes your jq has inputs and that you include -n as a command-line option.