How to remove filebeat tags like id, hostname, version, grok_failure message

Question

I am new to elk my sample log is look like

2017-01-05T14:28:00 INFO zeppelin IDExtractionService transactionId abcdef1234 operation extractOCRData received request duration 12344 exception error occured

my filebeat configuration is below

filebeat.prospectors:
- input_type: log
  paths:
    - /opt/apache-tomcat-7.0.82/logs/*.log

document_type: apache-access
fields_under_root: true

output.logstash:
  hosts: ["10.2.3.4:5044"]

And my logstash filter.conf file:

filter {
  grok {
    match => [ "message", "transactionId %{WORD:transaction_id} operation %{WORD:otype} received request duration %{NUMBER:duration} exception %{WORD:error}" ]
  }
}
filter {
    if "beats_input_codec_plain_applied" in [tags] {
        mutate {
            remove_tag => ["beats_input_codec_plain_applied"]
        }
    }
}

; In kibana dashboard i can see log output as below

beat.name:
    ebb8a5ec413b
beat.hostname:
    ebb8a5ec413b
host:
    ebb8a5ec413b
tags:
beat.version:
    6.2.2
source:
    /opt/apache-tomcat-7.0.82/logs/IDExtraction.log
otype:
    extractOCRData
duration:
    12344
transaction_id:
    abcdef1234
@timestamp:
    April 9th 2018, 16:20:31.853
offset:
    805,655
@version:
    1
error:
    error
message:
    2017-01-05T14:28:00 INFO zeppelin IDExtractionService transactionId abcdef1234 operation extractOCRData received request duration 12344 exception error occured
_id:
    7X0HqmIBj3MEd9pqhTu9
_type:
    doc
_index:
    filebeat-2018.04.09
_score:
    6.315 

1 First question is how to remove filebeat tag like id,hostname,version,grok_failure message

2 how to sort logs on timestamp basis because Newly generated logs not appearing on top of kibana dashboard

3 Is there any changes required in my grok filter

Answer 1

You can remove filebeat tags by setting the value of fields_under_root: false in filebeat configuration file. You can read about this option here.

If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. If the custom field names conflict with other field names added by Filebeat, the custom fields overwrite the other fields.

you can check if _grokparsefailure is in tags using, if "_grokparsefailure" in [tags] and remove it with remove_tag => ["_grokparsefailure"]

Your grok filter seems to be alright.

Hope it helps.