Search code examples
.net-corelogstashserilogfluentdelk

.NET Core API Serilog-ELK VS logstash/fluentd

the following are my understanding

  1. .net core api with serilog singk to ELK can directly send logs to ELK
  2. Logstash & Fluentd is needed only if we want to send a log file (by massaging the data) to ELK

my question is

  1. why do I need logstash | fluentd if I can directly send my logs to ELK using a serilog sink in my api?
  2. If I send using serilog sing to ELK directly what happens if the connection to ELK is down? will it save temporarily and re send?
  3. I read in article it says FluentD uses persistent queue and Logstash doesn't but why exactly this queue needed? lets say If my app have 1 logfile and it gets updated every second. So logstash sends the whole file to ELK every second? even if it fails it can resend my log file to ELK right? so why a persistent queue needed here for Fluentd/ logstash comparasion.

Appreciate some clear explanation on this.

Solution
    1. why do I need logstash | fluentd if I can directly send my logs to ELK using a serilog sink in my API?
    2. If I send using serilog sing to ELK directly what happens if the connection to ELK is down? will it save temporarily and re send?

    Question 2 answers question 1 here. FluentD has a battle-tested buffering mechanism to deal with ELK outages. Moreover, you don't want to use the app thread to deal with a task completely unrelated to an app - log shipping. This complicates your app and decreases portability.

    1. I read in article it says FluentD uses persistent queue and Logstash doesn't but why exactly this queue needed? lets say If my app have 1 logfile and it gets updated every second. So logstash sends the whole file to ELK every second? even if it fails it can resend my log file to ELK right? so why a persistent queue needed here for Fluentd/ logstash comparasion.

    Correct. FluentD has a buffer https://docs.fluentd.org/configuration/buffer-section. It will send whatever came for the period of time you've set in match (buffer is used to accumulate logs for the time period here). If the log backend (ELK) is down, it will keep storing the unsuccessful log records in the buffer. Depending on the buffer size this can handle pretty severe log backend outages. Once the log backend (ELK) is up again, all the buffers are sent to it and you don't lose anything.

    Logstash's persistent queue is a similar mechanism, but they went further and after the in-memory buffer they added external queues like Kafka. FluentD is also capable to use the queue when you use kafka input/output, and you still have a buffer on top of this in case a Kafka goes down.