logstash kv filter cannot convert strings to integers

Question

I have a log with a format similar to:

stage_name=stage1
stage_duration=30
stage_result=failed

The problem is I am using this parser on multiple log files with each basically containing numerous kv pairs.

Is there a way to recognize when values are integers and cast them as such without having to use mutate on every single key value pair? (Rather than a string)

i found that using ruby will might solve my issue by i need to understand how to replace the ' ' = space with /n = next row

input {
    stdin {}
}

filter {
    ruby {
        code => "
            fieldArray = event['message'].split(' ');
            for field in fieldArray
                name = field.split('=')[0];
                value = field.split('=')[1];
                if value =~ /\A\d+\Z/
                    event[name] = value.to_i
                else
                    event[name] = value
                end
            end
        "
    }
}
output {
    stdout { codec => rubydebug }
}

Answer 1

If mapping is not given when a field is created, the field type is automatically determined using ECS (Elastic Common Schema). The same goes for the Logstash kv filter Ref.

You can use dynamic template to determine the type of the field.

For example: If a field name is stage_duration or *_duration, let them be integers.

To set the dynamic template: Ref.

PUT hagay_bar/
{
   "mapping": {
     "dynamic_templates": [
       {
         "strings_as_integer": {
           "match_mapping_type": "string",
           "match": "*_duration",
           "mapping": {
           "type": "integer"
           }
         }
       }
     ]
   }
}

Note: The mapping of an index cannot be changed. A new index must be created.