Search code examples
jsonelasticsearchlogstashkibanafilebeat

Can Filebeat parse JSON fields instead of the whole JSON object into kibana?

I am able to get a single JSON object in Kibana:

enter image description here

By having this in the filebeat.yml file:

output.elasticsearch:
  hosts: ["localhost:9200"]

How can I get the individual elements in the JSON string. So say if I wanted to compare all the "pseudorange" fields of all my JSON objects. How would I:

  1. Select "pseudorange" field from all my JSON messages to compare them.
  2. Compare them visually in kibana. At the moment I can't even find the message let alone the individual fields in the visualisation tab...

I have heard of people using logstash to parse the string somehow but is there no way of doing this simply with filebeat? If there isn't then what do I do with logstash to help filter the individual fields in the json instead of have my message just one big json string that I cannot interact with?

I get the following output from output.console, note I am putting some information in <> to hide it:

  "@timestamp": "2021-03-23T09:37:21.941Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.8.14",
    "truncated": false
  },
  "message": "{\n\t\"Signal_data\" : \n\t{\n\t\t\"antenna type:\" : \"GPS\",\n\t\t\"frequency type:\" : \"GPS\",\n\t\t\"position x:\" : 0.0,\n\t\t\"position y:\" : 0.0,\n\t\t\"position z:\" : 0.0,\n\t\t\"pseudorange:\" : 20280317.359730639,\n\t\t\"pseudorange_error:\" : 0.0,\n\t\t\"pseudorange_rate:\" : -152.02620448094211,\n\t\t\"svid\" : 18\n\t}\n}\u0000",
  "source": <ip address>,
  "log": {
    "source": {
      "address": <ip address>
    }
  },
  "input": {
    "type": "udp"
  },
  "prospector": {
    "type": "udp"
  },
  "beat": {
    "name": <ip address>,
    "hostname": "ip-<ip address>",
    "version": "6.8.14"
  },
  "host": {
    "name": "ip-<ip address>",
    "os": {
      <ubuntu info>
    },
    "id": <id>,
    "containerized": false,
    "architecture": "x86_64"
  },
  "meta": {
    "cloud": {
      <cloud info>
    }
  }
}
Solution

  • In Filebeat, you can leverage the decode_json_fields processor in order to decode a JSON string and add the decoded fields into the root obejct:

    processors:
  - decode_json_fields:
      fields: ["message"]
      process_array: false
      max_depth: 2
      target: ""
      overwrite_keys: true
      add_error_key: false