I am trying to write logs from my .Net application to Elasticsearch using the serilog-elasticsearch sink . I was able to create indexes dynamically and write logs.
I was using the app.settings method for serilog-elastic. Following is my app-settings, in the .Net project from which I create logs.
<add key="serilog:using" value="Serilog.Sinks.Elasticsearch" />
<add key="serilog:write-to:Elasticsearch.nodeUris" value="http://localhost:9200" />
<add key="serilog:write-to:Elasticsearch.indexFormat" value="app-index-{0:yyyy.MM.dd}" />
<!--<add key="serilog:write-to:Elasticsearch.templateName" value="apptemplate" />-->
<add key="serilog:write-to:Elasticsearch.typeName" value="appLogEvent" />
<add key="serilog:write-to:Elasticsearch.pipelineName" value="geoip" />
<add key="serilog:write-to:Elasticsearch.batchPostingLimit" value="50" />
<add key="serilog:write-to:Elasticsearch.emitEventFailure" value="WriteToSelfLog" />
<add key="serilog:write-to:Elasticsearch.period" value="2" />
<add key="serilog:write-to:Elasticsearch.inlineFields" value="true" />
<add key="serilog:write-to:Elasticsearch.minimumLogEventLevel" value="Debug" />
<!--<add key="serilog:write-to:Elasticsearch.bufferBaseFilename" value="C:\Logs\elastic-buffer.log" />-->
<add key="serilog:write-to:Elasticsearch.bufferFileSizeLimitBytes" value="5242880" />
<add key="serilog:write-to:Elasticsearch.bufferLogShippingInterval" value="5000" />
<!--<add key="serilog:write-to:Elasticsearch.connectionGlobalHeaders" value="Authorization=Bearer SOME-TOKEN;OtherHeader=OTHER-HEADER-VALUE" />-->
<add key="logPath" value="C:\Logs\App" />
<add key="logLevel" value="Error" />
<!-- Log Rolling: File size limit for roll over (code defaults to 50MB if not specified here) -->
<add key="logRollFileSizeLimit" value="52428800" />
<!-- Log Rolling: Number of files to retain when rolled over (defaults to 10 if not specified here) -->
<add key="logRollFileRetainCount" value="10" />
I needed to apply a plugin geoip-processor on the indexed documents and identify the IP addresses in the log and convert them to geo-codes.
I created pipeline to execute processors and map the necessary data. I was able to test it using the dev-console tools. But when I applied the same pipeline on the documents by specifying in the app.settings, it does not work. I created different pipelines and tried, but with little success. Following is my pipeline:
{
"geoip" : {
"description" : "Add geo ip info",
"processors" : [
{
"set" : {
"field" : "IP1",
"value" : "{{response.IP}}"
}
},
{
"trim" : {
"field" : "IP1"
}
},
{
"geoip" : {
"field" : "IP1",
"target_field" : "geo_address",
"ignore_missing" : true,
"database_file" : "GeoLite2-Country.mmdb"
}
}
]
}
}
I checked the elastic logs, but nothing suspicious was found. The version of elasticsearch is: 6.6.0 .Net version is 4.5.2 Other version's of serilog related nuget packages from packages.config file are:
<package id="ElasticSearch.Net" version="5.5.0" targetFramework="net452" />
<package id="Serilog" version="2.5.0" targetFramework="net452" />
<package id="Serilog.Formatting.Compact" version="1.0.0" targetFramework="net452" />
<package id="Serilog.Settings.AppSettings" version="2.1.0" targetFramework="net452" />
<package id="Serilog.Sinks.ElasticSearch" version="5.4.0" targetFramework="net452" />
<package id="Serilog.Sinks.File" version="4.0.0" targetFramework="net452" />
<package id="Serilog.Sinks.PeriodicBatching" version="2.1.1" targetFramework="net452" />
<package id="Serilog.Sinks.RollingFile" version="3.3.0" targetFramework="net452" />
<package id="Serilog.Sinks.Seq" version="3.3.3" targetFramework="net452" />
The issue was because of the version conflict of elasticsearch and Serilog.Sinks.Elasticsearch. It seems that the version needed to be updated to the latest. I was using Serilog.Sinks.Elasticsearch version=5.4.0 and updated to the latest (7.1.0). The pipeline is now executing as intended. The ElasticSearch version was 6.6.0.
Unfortunately the version compatibility was never specified anywhere in the github page