logstash, plugin kv and duplicated value in field

Question

Example log:

2017-11-27T04:26:33+01:00 node231 PROXY-NODE2: 2017-11-27 04:26:31,559 [INFO] [user=optimus&id=dwql33333fssd&password=****&request=true&response=true&total=225&id=dwql33333fssd]

My logstash config:

filter {
    if "node" in [tags] and  "[INFO]" in [message] {
        grok {
            match => { "message" => "%{TIMESTAMP_ISO8601} %{DATA} %{DATA:[proxy][node]}: %{TIMESTAMP_ISO8601:[proxy][timestamp]} %{DATA} \[%{GREEDYDATA:data}\]"
            add_field => {"[@metadata][status]" => "parsed"}
        }
        if [@metadata][status] == "parsed" {
            kv {
             source => "data"
             field_split => "&"
             include_keys => [ "user", "id", "total" ]
        }
    }
}

Output(example):

{
    "id" => [
              [0] "dwql33333fssd",
              [1] "dwql33333fssd"
            ],
}

Because log contains twice "id"... how I can get only first or the last "id"? I need the flexible solution because sometimes other fields also duplicated.

Answer 1

A bool option for removing duplicate key/value pairs. When set to false, only one unique key/value pair will be preserved.

For example, consider a source like from=me from=me. [from] will map to an Array with two elements: ["me", "me"]. To only keep unique key/value pairs, you could use this configuration:

  filter {
      kv {
        allow_duplicate_values => false
      }
    }

Source: https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html#plugins-filters-kv-allow_duplicate_values