How to parse a fluent-bit json $log.<field> key?

Question

I have the following log to be parsed:

TID: [-1234] [] [2021-05-31 09:53:26,680] - Unique ID: Evento_Teste, Event: {"event":{"metaData":"blue"}}

And below the configuration files that I created to deal with it.

#td-agent-bit.conf
[SERVICE]
    log_level info
    parsers_file parsers.conf

[INPUT]
    Name tail
    Path /opt/wso2am/repository/logs/xtest-td.log
    Tag wso2.trace

[FILTER]
    name   parser
    match  wso2.*
    Key_Name log
    Parser wso2_trace

[OUTPUT]
    Name  stdout
    match *
    Format json_lines

# parsers.conf
[PARSER]
    Name   wso2_trace
    Format regex
    Regex  ^TID: \[(?<TID>[-\d]+)\] \[\] \[(?<time>[^\]]*)\].* Unique ID: (?<unique_id>[_\w\d]+), Event: (?<event>.*)$
    Time_Key time
    Time_Format %Y-%m-%d %H:%M:%S

And I'm getting the following in the output:

[2021/06/01 16:09:15] [ info] [input:tail:tail.0] inotify_fs_add(): inode=34902473 watch_fd=1 name=/opt/wso2am/repository/logs/xtest-td.log
{"date":1622454806.0,"TID":"-1234","unique_id":"Evento_Teste","event":"{\"event\":{\"metaData\":\"blue\"}}"}

This log will be sent to a ES instance, and I need to transform the value of the event key in an json object instead of a string as is.

I've tried to create some parsers using decoders but I couldn't find a way to transform the event in a json object.

How can I do it ?

Answer 1

I figured out how to parse.

I've created a new parse:

[PARSER]
    Name         json_field
    Format       json
    # Command       | Decoder      | Field    | Optional Action   |
    # ==============|==============|==========|===================|
    Decode_Field_As   escaped_utf8   log        do_next
    Decode_Field_As   json           log

And I included it in the pipeline:

[SERVICE]
    log_level info
    parsers_file parsers.conf

[INPUT]
    Name tail
    Path /opt/wso2am/repository/logs/xtest-td.log
    Tag wso2.trace

[FILTER]
    name   parser
    match  wso2.*
    Key_Name log
    Parser wso2_trace

[FILTER]
    name   parser
    match  wso2.*
    Key_Name event
    Parser json_field
    Reserve_Data True

[OUTPUT]
    Name  stdout
    match *
    Format json_lines