Search code examples
elasticsearchelastic-stackelastic-beats

What happens when the ouput of Auditbeat is down

I am using the following pipeline to forward data

Auditbeat ---> logstash ---> ES

Suppose if the logstash machine goes down, I want to know how the Auditbeat handles the situation.

I would like to know the specifics like

  1. is there a retry mechanism?
  2. how long will it retry?
  3. what happens to the audit logs, will it be lost?

  4. the reason that I ask question 3 is that, we enable auditbeat by disabling auditd service (which was generating the auditlogs under /var/log/audit/audit.log). SO if logstash goes down there is no data forwarding happening and hence there is a chance of data loss. Please clarify.

  5. if auditbeat is storing the data while logstash is down, where is it doing so? and what is the memory(disk space) allocated to this saving process?

Thanks in advance

Solution

  • Auditbeat has an internal queue which stores the events before sending it to the configured output, by default this queue is a memory queue that will store up to 4096 events.

    If the queue is full, no more events will be stored until the output comes back and start to receive data from auditbeat, there is a risk of data loss here.

    You can change the number of the events that the memory queue stores.

    There is also the option to use a file queue, which will save the events to disk before sending to the configured output, but this feature is still in beta.

    You can read about the internal queue in the documentation.