Customize log output in Kibana

Question

Finally, I got working ELK stack to get some logs from a remote server. However, I would like to customize the output of the logs. Is there a way to remove some fields which I am highlighting in yellow:

I tried to remove them from _source including remove_field in the logstash.conf:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/..."
    ssl_key => "/..logstash.key"
  }
}

filter {
        grok {
            match => {
                "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
            }
            remove_field => [ "tags", "prospector.type", "host.architecture", "host.containerized", "host.id", "host.os.platform", "host.os.family" ]
        }
}

output {
    elasticsearch {
        hosts => "localhost:9200"
        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    }
}

Do you know how can I get rid of the yellow fields in _source for the logs coming from filebeat?

Update of logstash.conf based on Leandro comments:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => ".../logstash.crt"
    ssl_key => ".../logstash.key"
  }
}

filter {
        grok {
            match => {
                "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
            }
            remove_field => [ "tags","[prospector][type]","[host][architecture]", "[host][containerized]", "[host][id]", "[host][os][platform]", "[host][os][family]", "[beat][hostname]", "[beat][name]", "[beat][version], "[offset]", "[input][type]", "[meta][cloud][provider]", "[meta][cloud][machine_type]", "[meta][cloud][instance_id]"]
        }
}



output {
    elasticsearch {
        hosts => "localhost:9200"
        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    }
}

In logs:

019-02-27T17:03:41.637-0800    DEBUG   [input] file/states.go:68       New state added for /logs/api.log
2019-02-27T17:03:41.637-0800    DEBUG   [registrar]     registrar/registrar.go:315      Registrar state updates processed. Count: 1
2019-02-27T17:03:41.637-0800    DEBUG   [registrar]     registrar/registrar.go:400      Write registry file: /filebeat/registry
2019-02-27T17:03:41.637-0800    INFO    log/harvester.go:255    Harvester started for file: /logs/api.log
2019-02-27T17:03:41.647-0800    DEBUG   [publish]       pipeline/processor.go:308       Publish event: {
  "@timestamp": "2019-02-28T01:03:41.647Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.6.0"
  },
  "log": {
    "file": {
      "path": "/logs/api.log"
    }
  },
  "input": {
    "type": "log"
  },
  "host": {
    "name": "tomcat",
    "os": {
      "family": "redhat",
      "name": "CentOS Linux",
      "codename": "Core",
      "platform": "centos",
      "version": "7 (Core)"
    },
    "id": "6aaed308aa5a419f880c5e45eea65414",
    "containerized": true,
    "architecture": "x86_64"
  },
  "meta": {
    "cloud": {
      "region": "CanadaCentral",
      "provider": "az",
      "instance_id": "6452bcf4-7f5d-4fc3-9f8e-5ea57f00724b",
      "instance_name": "tomcat",
      "machine_type": "Standard_D8s_v3"
    }
  },
  "message": "2018-09-14 20:23:37 INFO  ContextLoader:272 - Root WebApplicationContext: initialization started",
  "source": "/logs/api.log",
  "offset": 0,
  "prospector": {
    "type": "log"
  },
  "beat": {
    "hostname": "tomcat",
    "version": "6.6.0",
    "name": "tomcat"
  }
}

Thanks

Answer 1

Some of those fields are nested fields, the way to access them in a Logstash filter is using the [field][subfield] notation.

Your remove_field shoud be something like this:

remove_field => ["tags","[host][architecture]","[meta][cloud][provider]"]

But I don't think you can remove the @version field.

UPDATE:

Using the event example from your Filebeat log I simulated a pipeline and got a _grokparsefailure, to remove the fields even when the grok fails you need to use the remove_field inside a mutate filter:

filter {
  grok {
     your grok
  }
  mutate {
    remove_field => ["[prospector]","[host][architecture]", "[host][containerized]", "[host][id]", "[host][os][platform]", "[host][os][family]", "[beat]", "[offset]", "[input]", "[meta]"]
  }
}

Don't remove the tags field until you have fixed your groks.

The logstash output on that example is:

{
  "source": "/logs/api.log",
  "tags": [
    "_grokparsefailure"
  ],
  "@timestamp": "2019-02-28T01:03:41.647Z",
  "message": "2018-09-14 20:23:37 INFO  ContextLoader:272 - Root WebApplicationContext: initialization started",
  "log": {
    "file": {
      "path": "/logs/api.log"
    }
  },
  "@version": "1",
  "host": {
    "os": {
      "codename": "Core",
      "version": "7 (Core)",
      "name": "CentOS Linux"
    },
    "name": "tomcat"
  }
}