Which server logging solution is less processor consuming among Kafka, Fluentd and Logstash?

Question

I have a distributed logging system to monitor load-balanced server entities. It is basic for me, that the server does not invest a lot of processor time in the logging process, allowing the application to run with the maximum resources possible.

It would be nice to know, which of those alternatives is "cheaper" in terms of processor time or, in case, to be recommended any other solution for this matter.

Answer 1

First of all, Kafka is not a log collector. It's a distributed message queue and can work with log collectors like Fluentd and Logstash both as consumers and producers.

Instead of opinions, let's put some numbers.

1. CPU Usage: This depends on how much filtering and data processing you do on the client-side using Fluentd and/or Logstash. If you do minimal processing, both can process 10,000+ messages per second. Both can take advantage of multiple CPUs (http://docs.fluentd.org/articles/in_multiprocess, for example)
2. Memory: Fluentd uses about 40MB of memory and Logstash uses about 100MB of memory. If this is too much, Logstash has Beats and Fluentd as Fluentd Forwarders (https://github.com/fluent/fluentd-forwarder).