Removing unwanted fields in Logstash configuration file

Question

I'm building a ELK Setup and its working fine , however i'm getting into a situation where i want to remove certain fields from by system-log data while processing through logstash like remove_field & remove_tag which i've defined in my logstash configuration file but that's not working.

Looking for any esteem and expert advice to correct the config to make it running, thanks very much in advanced.

My logstash configuration file:

[root@sandbox-prd~]# cat /etc/logstash/conf.d/syslog.conf
input {
  file {
    path => [ "/data/SYSTEMS/*/messages.log" ]
    start_position => beginning
    sincedb_path => "/dev/null"
    max_open_files => 64000
    type => "sj-syslog"
 }
}

filter {
  if [type] == "sj-syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp } %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      remove_field => ["@version", "host", "_type", "_index", "_score", "path"]
      remove_tag => ["_grokparsefailure"]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
  }
 }
}
output {
        if [type] == "sj-syslog" {
        elasticsearch {
                hosts => "sandbox-prd02:9200"
                manage_template => false
                index => "sj-syslog-%{+YYYY.MM.dd}"
                document_type => "messages"
  }
 }
}

Data sample appearing on the Kibana Portal

syslog_pid:6662 type:sj-syslog syslog_message:(root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) syslog_severity:notice syslog_hostname:dbaprod01 syslog_severity_code:5 syslog_timestamp:Feb 11 10:25:02 @timestamp:February 11th 2019, 23:55:02.000 message:Feb 11 10:25:02 dbaprod01 CROND[6662]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) syslog_facility:user-level syslog_facility_code:1 syslog_program:CROND received_at:February 11th 2019, 10:25:03.353 _id:KpHo2mgBybCgY5IwmRPn _type:messages
_index:sj-syslog-2019.02.11 _score: -

MY Resource Details:

OS version : Linux 7

Logstash Version: 6.5.4

Answer 1

You can't remove _type and _index, those are metadata fields needed by elasticsearch to work, they have information about your index name and the mapping of your data, the _score field is also a metadata field, generated at search time, it's not on your document.