How to parse a mixed custom log using filebeat and processors

Question

I'm trying to parse a custom log using only filebeat and processors. I wouldn't like to use Logstash and pipelines.

Below a sample of the log:

TID: [-1234] [] [2021-08-25 16:25:52,021]  INFO {org.wso2.carbon.event.output.adapter.logger.LoggerEventAdapter} - Unique ID: Evento_Teste, Event: {"event":{"host":"example.com","server":"WSO2 API Manager"}}

Then, I need to get the date 2021-08-25 16:25:52,021 and make it my _doc timestamp and get the Event and make it my message.

After many tries I'm only able to dissect the log using the following configuration:

filebeat.inputs:

- type: log

  enabled: true
  paths:
    - /tmp/a.log
  processors:
    - dissect:
        tokenizer: "TID: [-1234] [] [%{@timestamp}]  INFO {org.wso2.carbon.event.output.adapter.logger.LoggerEventAdapter} - Unique ID: Evento_Teste, Event: %{event}"
        field: "message"

output.console:
  pretty: true

And getting the following as output:

{
  "@timestamp": "2021-08-25T19:58:00.525Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.12.1"
  },
  "input": {
    "type": "log"
  },
  "dissect": {
    "@timestamp": "2021-08-25 16:25:52,021",
    "event": "{\"event\":{\"host\":\"example.com\",\"server\":\"WSO2 API Manager\"}}"
  },
  "host": {
    "name": "dtrsrvhomapim301"
  },
  "agent": {
    "ephemeral_id": "1555da2b-234f-444e-a0fe-42b49fb73b38",
    "id": "1b43e769-87be-4087-9876-70281ceb3cf5",
    "name": "dtrsrvhomapim301",
    "type": "filebeat",
    "version": "7.12.1",
    "hostname": "dtrsrvhomapim301"
  },
  "ecs": {
    "version": "1.8.0"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/tmp/a.log"
    }
  },
  "message": "TID: [-1234] [] [2021-08-25 16:25:52,021]  INFO {org.wso2.carbon.event.output.adapter.logger.LoggerEventAdapter} - Unique ID: Evento_Teste, Event: {\"event\":{\"host\":\"example.com\",\"server\":\"WSO2 API Manager\"}}"
}

I couldn't figure out how to make the dissect.@timestamp as my @timestamp, and how to parse the dissect.event as a json and make it my message.

How could those be done ?

Answer 1

You can avoid the "dissect" prefix by using target_prefix: "" . Json fields can be extracted by using decode_json_fields processor. You might want to use a script to convert ',' in the log timestamp to '.' since parsing timestamps with a comma is not supported by the timestamp processor. The target field for timestamp processor is @timestamp by default

 processors:
    - dissect:
        tokenizer: "TID: [-1234] [] [%{time}]  INFO {org.wso2.carbon.event.output.adapter.logger.LoggerEventAdapter} - Unique ID: Evento_Teste, Event: %{event}"
        field: "message"
        target_prefix: ""
        overwrite_keys: true
    - decode_json_fields:
        fields: ["event"]
        process_array: false
        max_depth: 2
        target: ""
        overwrite_keys: true
        add_error_key: true
    - script:
        lang: javascript
        source: >
          function process(evt) {
            var ts = evt.Get('time').replace(',', '.');
            evt.Put('time', ts);
          }
    - timestamp:
        field: "time"
        layouts:
          - '2006-01-02 15:04:05.999'
  output.console:
    pretty: true