how to remove field in logstash output

Question

I have set up an ELK stack. For the logstash instance, it has two output including Kafka and elasticsearch.

For the output of elasticsearch, I want to keep the field @timestamp. For the output of Kafka, I want to remove the field @timestamp. So I cannot just remove field @timestamp in the filter. I just want it removed for the Kafka output.

I have not found this kind of solution.

append

Try to use clone plugin:

clone {
  clones => ["kafka"]
  id => ["kafka"]
  remove_field => ["@timestamp"]
}

output {

if [type] != "kafka" {
  elastcsearch output
}

if [type] == "kafka" {
  kafka output
}
}

It's strange that the output of elasticsearch can work. But it cannot output to kafka. And I have tried to judge by id, still does not wordk.

Answer 1

Since you can only remove fields in the filter block, to have the same pipeline output two different versions of the same event you will need to clone your events, remove the field in the cloned event and use conditionals in the output.

To clone your event and remove the @timestamp field you will need something like this in your filter block.

filter {
    # your other filters
    #
    clone {
        clones => ["kafka"]
    }
    if [type] == "kafka" {
        mutate {
            remove_field => ["@timestamp"]
        }
    }
}

This will clone the event and the cloned event will have the value kafka in the field type, you will then use this field in the conditionals in your output.

output {
    if [type] != "kafka" {
        your elasticsearch output
    }
    if [type] == "kafka" {
        your kafka output
    }
}