Logstash difference between add_field and replace

Question

What is the difference between add_field and replace, when configuring Logstash?

Semantically, they might expected to do different things, but the current manual does not have anything to say about how the two functions differ depending on the pre-existance of the relevant fields.

Example 1:

filter {
  mutate {
    add_field => { "foo_%{somefield}" => "Hello world, from %{host}" }
  }
}

The manual does not say what the expected behavour is if the field already exists. Is the behaviour the same as replace?

Example 2:

filter {
  mutate {
    replace => { "message" => "%{source_host}: My new message" }
  }
}

Once again the manual does not say what the expected behaviour is if the field does not already exist. Is it the same as add_field?

Answer 1

Semantically, they might seem equivalent but they are different as add_field might not always kick in.

replace is part of the mutate filter and if you're looking at the source code of that plugin, you'll see that replace will always set a field even if it doesn't exist:

  def replace(event)
    @replace.each do |field, newvalue|
      event.set(field, event.sprintf(newvalue))
    end
  end

add_field, however, is a generic configuration of any input and filter plugins and you'll often see in the documentation of some specific filter plugins something like this (e.g. for the grok filter):

If this filter is successful, add any arbitrary fields to this event. [...]

This configuration is inherited from LogStash::Filters::Base, which is the superclass of all filter plugins. This means that in certain filter plugins, in order for add_field to actually add the field, the filter must be successful, otherwise no fields is added.

So in your case, mutate/replace will always set a field even if it doesn't exist, and mutate/add_field will add the specified fields after running all the operations of the mutate filter.

So if you have a mutate filter that does some specific operation like gsub and fails for some reason, no fields will be added as a result.