I am trying to send my logs from .netcore 3.1 API to Loki server. For doing that I'm using Serilog-Sinks-Loki nugget package. When I run my code, logs are sent immediately and I can see them in Grafana. As time past, logs are coming with delay which growing bigger and bigger. Eventually I see my logs arriving after delay of 10 minutes to Loki. I read about that and understand that Serilog-Sinks-Loki sits on top of Serilog-Sinks-Http so I need to configure something over there to make my logs arrive faster but not sure how, can someone help me with that please? My goal is to send my logs immediately, with no delay.
Here is my program.cs code
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.UseLamar()
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
})
.ConfigureLogging(logging =>
{
var credentials = new BasicAuthCredentials("LOKI-SERVER-URL", "USER", "PASSWORD");
var log = new LoggerConfiguration()
.MinimumLevel.Verbose()
.Enrich.FromLogContext()
.WriteTo.LokiHttp(credentials)
.CreateLogger();
logging.AddSerilog(log);
});
}
Although Serilog.Sinks.Loki wraps a Serilog.Sinks.Http sink, it doesn't expose any of the settings that would allow you to configure how often messages are sent (batchPostingLimit, period, etc.)
That said, the default period of Sinks.Sink.Http is 2 seconds, thus 10 minutes is too long for messages to be sent with the default settings. You're likely having connection issues that are blocking the sink from sending the messages or blocking the server from receiving them.
Have you tried enabling the SelfLog to see if any errors are being caught by the sink?
If not, you can monitor the network traffic to see if you can find failed attempts and/or timeouts when the sink tries to send the log events.
You can also extend the LokiHttpClient and write some debug messages to a file or something to see how often / how many messages are being sent.
For reference here is a recent issue in the Serilog.Sinks.Loki repo (perhaps you created it?)