Hey I am working with fluentd recently. So the thing is I am making an environment where fluentd from different servers will send logs to central system with fluentd installed where the processing of the logs will be done.
But I am stuck at this point. I'm unable to understand the mechanism it uses to transfer logs from one node to another. Also I want to use some messaging queue like rabbitmq here to do this for me.
I'm unable to find suitable answer for this question on google and i dont find the documentation very suitable for me to understand everything about fluentd and its plugins. and since my networking concepts are not that strong, some things go over my head. Also I found out that it sends logs to treasure-data by default and there is a way to disable that. I want some understanding on how to use rabbitmq to do this for me.
@repeatedly Thanks for your efforts. I already did the reading of the documentation. Its documentation needs some more work.
Found the solution. fluentd collector sends the log directly using tcp protocol. I have configured my fluentd to use kafka as a messaging queue. fluentd collector collects the logs from the log files and send them to the kafka server from where they should be sent to elasticsearch. So used another fluentd layer to collect logs from kafka and forward them to elasticsearch. It works fine.