Use logstash to analize log line

Question

Hi I want to implement els stack for better log review, so I added logstash to ealsticsearch and kibana that already exists, logs are in form of files that was generated by .net apps that uses serilog (I know I can use serilog sink to add it directly to elasticsearch).

So I have a logstash configuration like this:

input 
{
    file 
    {
        mode => "tail"
        path => "/usr/share/logstash/ingest_data/*"
    }
}

filter {
}

output 
{
    elasticsearch 
    {
        index => "logstash-%{+YYYY.MM.dd}"
        hosts=> "${ELASTIC_HOSTS}"
        user=> "${ELASTIC_USER}"
        password=> "${ELASTIC_PASSWORD}"
        cacert=> "certs/ca/ca.crt"
    }
}

But I have a simple log line:

2023-08-17 10:10:47.253 +02:00 [Warning] [Application] Logged message

Now I see in Data View he sees timestamp source and some other default tags, but is there a possibility to define filter in such way he add information lvl like [Warning] and source as [Application]?

Answer 1

You can use a Grok filter. You can find the already defined pattern here.

2023-08-17 10:10:47.253 +02:00 [Warning] [Application] Logged message

can be parsed with

%{DATESTAMP:time} %{ISO8601_TIMEZONE:timezone} [%{WORD:level}] [%{WORD:source}] %{GREEDYDATA:logMessage}

where the syntax is %{PATTERN_NAME:fieldName}.