Search code examples
jsonloggingelastic-stackfilebeat

Filebeat 7.3.0 not parsing multiline json

I started with ELK 7.3.0 for SIEM feature. I now need to integrate Cloudflare logs which is in JSON format. I'm pretty sure that filebeat can do that.

I have installed filebeat rpm package on my CentOS 7 Server

What I have done so far...

Elastic port changed to : 1300 Logstash port changed to: 2300 Kibana port changed to: 3300

filebeat.yml as follows

filebeat.config.inputs:
   enabled: true
   path: configs/*.yml
   reload.enabled: true
   reload.period: 10s

setup.kibana: 
   host: "10.139.111.216:3300"

output.elasticsearch:
   hosts: ["10.139.111.216:1300"]

I have placed my config in /etc/filebeat/modules.d/cloudflare-FW.conf directory as this will reload if any config changes made as I have mentioned in filebeat.yml

cloudflare-FW.conf

filebeat.inputs:
- type: log
  enabled: true
  paths:
  - /var/crons/logs/*.json
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match:  after

processors:
- decode_json_fields:
 fields: ['message']
 target: json

output.elasticsearch:
  hosts: ["10.136.111.216:1300"]
  index: "cloudflare-FW-%{+yyyy.MM.dd}"
  setup.template.enabled: true
  setup.template.name: "cloudflare-FW"
  setup.template.pattern: "cloudflare-FW-*"

This is my target json file to parse

{
  "ray_id": "501dxxxxxxxx0ce1b",
  "kind": "firewall",
  "source": "bic",
  "action": "drop",
  "rule_id": "bic",
  "ip": "161.00.6.8",
  "ip_class": "noRecord",
  "country": "GB",
  "colo": "LHR",
  "host": "site.com",
  "method": "POST",
  "proto": "HTTP/1.1",
  "scheme": "https",
  "ua": "Jakarta Commons-HttpClient/3.1",
  "uri": "/nxxxxxxx/senxxxxxxxxail.php",
  "matches": [
    {
      "rule_id": "bic",
      "source": "bic",
      "action": "drop"
    }
  ],
  "occurred_at": "2019-08-06T03:28:57Z"
},
{
  "ray_id": "501dxxxxxxxxxx6ce27",
  "kind": "firewall",
  "source": "bic",
  "action": "drop",
  "rule_id": "bic",
  "ip": "161.61.6.8",
  "ip_class": "noRecord",
  "country": "GB",
  "colo": "LHR",
  "host": "site.com",
  "method": "POST",
  "proto": "HTTP/1.1",
  "scheme": "https",
  "ua": "Jakarta Commons-HttpClient/3.1",
  "uri": "/nxxxxxxx/senxxxxxxxxail.php",
  "matches": [
    {
      "rule_id": "bic",
      "source": "bic",
      "action": "drop"
    }
  ],
  "occurred_at": "2019-08-06T03:02:52Z"
}

When I run filebeat -e it should create a index named cloudflare-FW-%{+yyyy.MM.dd} and should ingest the data from the json file to the index. In the end JSON key should be the fields which can be queried on.

But it is not happening. Any help is greatly appreciated. Keep in mind this is filebeat 7.3.0

Thanks

Solution

  • I have placed my config in /etc/filebeat/modules.d/cloudflare-FW.conf directory

    In your filebeat.yml you specified a totally different path:

    filebeat.config.inputs:
  enabled: true
  path: "configs/*.yml" <--
  reload.enabled: true
  reload.period: 10s

    Seems to me like your configuration is not even loaded because its in the wrong location.

    Also notice the file extension you specified (*.yml).