Fluent-bit - Splitting json log into structured fields in Elasticsearch

Question

I am trying to find a way in Fluent-bit config to tell/enforce ES to store plain json formatted logs (the log bit below that comes from docker stdout/stderror) in structured way - please see image at the bottom for better explanation. For example, apart from (or along with) storing the log as a plain json entry under log field, I would like to store each property individually as shown in red.

The documentation for Filters and Parsers are really poor and not clear. On top of that the forward input doesn't have a "parser" option. I tried json/docker/regex parsers but no luck. My regex is here if I have to use regex. Currently using ES (7.1), Fluent-bit (1.1.3) and Kibana (7.1) - not Kubernetes.

If anyone can direct me to an example or give one I would be much appreciated.

Thanks

{
  "_index": "hello",
  "_type": "logs",
  "_id": "T631e2sBChSKEuJw-HO4",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-06-21T21:34:02.000Z",
    "tag": "php",
    "container_id": "53154cf4d4e8d7ecf31bdb6bc4a25fdf2f37156edc6b859ba0ddfa9c0ab1715b",
    "container_name": "/hello_php_1",
    "source": "stderr",
    "log": "{\"time_local\":\"2019-06-21T21:34:02+0000\",\"client_ip\":\"-\",\"remote_addr\":\"192.168.192.3\",\"remote_user\":\"\",\"request\":\"GET / HTTP/1.1\",\"status\":\"200\",\"body_bytes_sent\":\"0\",\"request_time\":\"0.001\",\"http_referrer\":\"-\",\"http_user_agent\":\"curl/7.38.0\",\"request_id\":\"91835d61520d289952b7e9b8f658e64f\"}"
  },
  "fields": {
    "@timestamp": [
      "2019-06-21T21:34:02.000Z"
    ]
  },
  "sort": [
    1561152842000
  ]
}

Thanks

conf

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    debug
    Parsers_File parsers.conf

[INPUT]
    Name   forward
    Listen 0.0.0.0
    Port   24224

[OUTPUT]
    Name  es
    Match hello_*
    Host  elasticsearch
    Port  9200
    Index hello
    Type  logs
    Include_Tag_Key On
    Tag_Key tag

Answer 1

Solution is as follows.

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    debug
    Parsers_File parsers.conf

[INPUT]
    Name         forward
    storage.type filesystem
    Listen       my_fluent_bit_service
    Port         24224

[FILTER]
    Name         parser
    Parser       docker
    Match        hello_*
    Key_Name     log
    Reserve_Data On
    Preserve_Key On

[OUTPUT]
    Name            es
    Host            my_elasticsearch_service
    Port            9200
    Match           hello_*
    Index           hello
    Type            logs
    Include_Tag_Key On
    Tag_Key         tag

[PARSER]
    Name         docker
    Format       json
    Time_Key     time
    Time_Format  %Y-%m-%dT%H:%M:%S.%L
    Time_Keep    On
    # Command      |  Decoder | Field | Optional Action
    # =============|==================|=================
    Decode_Field_As   escaped_utf8    log    do_next
    Decode_Field_As   json       log