logstashlogstash-grok
Grok can't create optional fields
I'm trying to parse logs in logstash with use of grok. In error log there are two more optional fields like requestUrl and requestMethod. but when i use grok for parsing logs to logstash.these from the error log grok can't create these two fields.
mypipeline.conf :
input {
beats {
type=>"mytest"
port => 5044
}
}
filter{
if [fields][log_type] == "gbase"
{
if [level] in [ "Error", "Fatal" ]
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:timestamp} %{INT:processId} %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{URI:requestUrl} %{USER:requestMethod} %{GREEDYDATA:message}"]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
else
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:timestamp} %{INT:processId} %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{GREEDYDATA:message}" ]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
}
if [fields][log_type] == "finance"
{
if [level] in [ "Error", "Fatal" ]
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:receivedAt} \[%{INT:processId}]\ %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{URI:requestUrl} %{WORD:requestMethod} %{GREEDYDATA:message}"]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
else
{
grok { match=> ["message","%{TIMESTAMP_ISO8601:receivedAt} \[%{INT:processId}]\ %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{GREEDYDATA:message}"]
overwrite => [ "message" ]}
mutate { gsub => ["message", "\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{4} ",""]}
mutate { gsub => ["message", "%{level}",""]}
mutate { gsub => ["message", "%{logger}",""]}
mutate { gsub => ["message", "%{clientIp}",""]}
mutate { gsub => ["message", "%{processId}",""]}
}
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
#match => [ "time" , "dd/MMM/yyyy:HH:mm:ss Z" ]
target=> "@timestamp"
}
}
output {
if [fields][log_type] == "finance"
{
elasticsearch
{
hosts => ["http://localhost:9200"]
index => "finance-%{+YYYY.MM.dd}"
user => "something"
password => "something"
}
}
if [fields][log_type] == "gbase"
{
elasticsearch
{
hosts => ["http://localhost:9200"]
index => "gbase-%{+YYYY.MM.dd}"
user => "something"
password => "something"
}
}
stdout { codec => rubydebug }
}
filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- D:\Git\gbase.API\API\Logs\*.log
tags: ["gbaseapi"]
fields:
log_type: gbase
multiline.pattern: ^\d\d\d\d-\d\d-\d\d
multiline.negate: true
multiline.match: after
multiline.max_lines : 1000
- type: log
enabled: true
paths:
- D:\Git\finance.api\FinanceAPI\logs\*.log
tags: ["financeapi"]
fields:
log_type: finance
multiline.pattern: ^\d\d\d\d-\d\d-\d\d
multiline.negate: true
multiline.match: after
multiline.max_lines : 1000
my log sample :
2020-10-10 21:14:59.8746 22152 WARN Validation bhavin 192.168.43.244 Sensitive data logging is enabled. Log entries and exception messages may include sensitive application data, this mode should only be enabled during development.
2020-10-10 21:15:35.2745 22152 DEBUG QuartzSchedulerThread bhavin 192.168.43.244 Batch acquisition of 0 triggers
2020-10-10 21:15:35.2783 22152 ERROR DeveloperExceptionPageMiddleware bhavin 192.168.43.244 http://bhavin/Leads/New POST An unhandled exception has occurred while executing the request.System.NullReferenceException: Object reference not set to an instance of an object.
json document
{
"input" => {
"type" => "log"
},
"processId" => "34076",
"log" => {
"file" => {
"path" => "D:\\Git\\gbase.API\\API\\Logs\\log-2020-10-10.log"
},
"flags" => [
[0] "multiline"
],
"offset" => 43633
},
"clientIp" => "192.168.43.244",
"fields" => {
"log_type" => "gbase"
},
"timestamp" => "2020-10-10 21:29:00.1492",
"ecs" => {
"version" => "1.5.0"
},
"user" => "bhavin",
"agent" => {
"name" => "Bhavin",
"hostname" => "Bhavin",
"type" => "filebeat",
"version" => "7.9.2",
"id" => "b5decaac-61a3-4dbc-860b-4423a0379391",
"ephemeral_id" => "208d7c9d-bdcd-45e9-8f82-ccdb08292203"
},
"level" => "ERROR",
"message" => "http://bhavin/Leads/New POST An unhandled exception has occurred while executing the request.GurukulERP.Logic.InvalidDataException: Provide a valid PhoneNumber.\n at GurukulERP.API.Controllers.CRM.LeadsController.New(AdmissionLead model) in D:\\Git\\gbase.API\\API\\Controllers\\CRM\\LeadsController.cs:line 515\n at lambda_method(Closure , Object , Object[] )\n at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\n at Microsoft.AspNetCore.Diagnostics..Invoke(HttpContext context) at GurukulERP.API.Controllers.CRM.LeadsController.New(AdmissionLead model) in D:\\Git\\gbase.API\\API\\Controllers\\CRM\\LeadsController.cs:line 515\n at lambda_method(Closure , Object , Object[] )\n at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()\n--- End of stack trace from previous location where exception was thrown ---\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\n at Microsoft.AspNetCore.Diagnostics..Invoke(HttpContext context)",
"host" => {
"name" => "Bhavin",
"hostname" => "Bhavin",
"architecture" => "x86_64",
"mac" => [
[0] "04:92:26:4a:aa:7b",
[1] "52:3e:aa:6a:59:4c",
[2] "50:3e:aa:6a:59:4c",
[3] "00:ff:73:c0:8e:ca",
[4] "50:3e:aa:6a:59:4c"
],
"id" => "2540a614-ed14-468c-82b6-3c2f1de5b687",
"ip" => [
[ 0] "fe80::8511:1aea:78ea:8722",
[ 1] "169.254.135.34",
[ 2] "fe80::a1dc:1a7d:8369:4ee5",
[ 3] "169.254.78.229",
[ 4] "fe80::c9b5:254f:10b3:191d",
[ 5] "169.254.25.29",
[ 6] "fe80::ac32:76b1:509a:bc5e",
[ 7] "169.254.188.94",
[ 8] "2405:204:808b:8dbb:d456:182e:dd87:ca6",
[ 9] "2405:204:808b:8dbb:3132:508b:53c1:132e",
[10] "fe80::d456:182e:dd87:ca6",
[11] "192.168.43.244"
],
"os" => {
"name" => "Windows 10 Enterprise",
"platform" => "windows",
"kernel" => "10.0.18362.1082 (WinBuild.160101.0800)",
"version" => "10.0",
"family" => "windows",
"build" => "18362.1082"
}
},
"@version" => "1",
"type" => "mytest",
"@timestamp" => 2020-10-10T15:59:02.229Z,
"tags" => [
[0] "gbaseapi",
[1] "beats_input_codec_plain_applied",
[2] "_dateparsefailure"
],
"logger" => "DeveloperExceptionPageMiddleware"
}
here is image of kibana view :
you can see the other fields are created but these two are not create
Solution
The field level does not exist in your document when it enters your pipeline, so the conditional you are using in your grok will never match to true.
Looking at your example messages, they have basically the same structure, they only start to be different after the ip address, you can use just one grok to parse all those messages and then have another grok that will only be applied if the overwritten message field starts with an URI.
The following filter will parse all of your example messages.
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{INT:processId} %{LOGLEVEL:level} %{DATA:logger} %{USER:user} %{IPV4:clientIp} %{GREEDYDATA:message}" }
overwrite => "message"
}
if [message] =~ "^http:" {
grok {
match => { "message" => "%{URI:requestUrl} %{USER:requestMethod} %{GREEDYDATA:message}" }
overwrite => "message"
}
}
}
The first grok will parse the messages and overwrite the message field with everything the comes after the ip address. So, in the case of the message that you want to extract the requestUrl and requestMethod, you will have the message field with the following value.
http://bhavin/Leads/New POST An unhandled exception has occurred while executing the request.System.NullReferenceException: Object reference not set to an instance of an object.
The if [message] =~ "^http:" part will check if the message field starts with http:, and if true it will parse this message, which will then create the fields requestUrl and requestMethod and a new overwritten message field.
Also, your gsub filters makes no sense, you can remove it.
You can apply the same process to your finance log type, the only difference seems to be that the processId appears between square brackets and you are using another field for the timestamp.
- How can I check nulls in Logstash pipeline in filter plugin?
- How to input a JSON file (array form) in Logstash?
- Log4J JsonTemplateLayout replacing message content
- logstash error: java.net.SocketException: Connection reset
- logstash keep both raw data and aggregated data
- Logstash Elasticsearch output plugin creates a single document index in Elasticsearch
- Index pattern is not showing up in Kibana management
- Problems with updating :sql_last_value
- Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit
- Logstash main class JvmOptionsParser not found / cannot load
- How to Send Structured Logs to ELK Stack with Custom Fields Using Python Logging?
- Querying keyword field in Logstash
- ELK index name doesn't change on rollover on the next day
- Grok pattern for [Mon Jan 04 08:36:12 2021]
- Ingesting SQL Server ErrorLog using Filebeat with the mssql module
- logstash unable to connect to elasticsearch
- How to discard the Duplicate values in ElasticSearch using DSL Query?
- Logstash Expected one of
- How can I manually trigger timing jdbc input in logstash?
- Error: exec /usr/local/bin/docker-entrypoint: permission denied for Logstash container on Podman
- Elasticsearch - Logstash Grok new field date formate is string and not date
- http.host: 0.0.0.0 - ERROR lookup logstash : no such host
- How to have an input of type MongoDB for Logstash
- Syncing PostgreSQL with ElasticSearch
- Problem with Logstash-Plug in-JMS and Weblogic 12.2.1
- Logstash Split Issue
- Logstash not responding at container image of kibana and logstash
- How to mention certificate in logstash/eck on kubernetes
- Kafka Connect Logstash
- How to add field for logsstream from specific name in elasticserach logstream