Adding a copy field and set it lowercase

Question

I'm having a trouble with a mutate filter, where i need to set a field copy to lowercase. But when i try to load on elastic , the field copy won't be lowercase.

My Logstash Version is 5.2

i'm using the configuration below:

filter {   
    mutate {
        rename => {"desc_ufe" => "state" }
        rename => {"desc_local" => "city" }
        rename => {"desc_bairro" => "neighborhood" }
        rename => {"desc_lograd" => "streetName" }
        rename => {"desc_cep" => "postalCode" }
        rename => {"desc_lograd_complemento" => "supplement" }
        rename => {"cod_mun" => "IBGEcode" }

        convert => {"IBGEcode" => "string"}

        add_field => {"statecp" => "%{state}"}
        add_field => {"citycp" => "%{city}"}
        add_field => {"neighborhoodcp" => "%{neighborhood}"}
        add_field => {"streetNamecp" => "%{streetName}"}
        add_field => {"supplementcp" => "%{supplement}"}

        lowercase => ["statecp","citycp","neighborhoodcp","streetNamecp","supplementcp"]
    }
 }

Thank you for any help.

Answer 1

You are falling prey to the order that the mutate filter does things. It's not based on the order in your config file -- it's based on the order that the mutate.rb code does it.

The order is (according to https://github.com/logstash-plugins/logstash-filter-mutate/blob/master/lib/logstash/filters/mutate.rb):

• coerce
• rename
• update
• replace
• convert
• gsub
• uppercase
• capitalize
• lowercase
• strip
• remove
• split
• join
• merge
• copy

Followed by anything else that applies to matching events (ie add_field)

Just move your lowercase to a 2nd mutate filter and it should work correctly.