I would like to configure filebeat to harvest stdout.log
to multiple elasticsearch outputs, say, lines with x==A
would be harvested to elasticsearchA
, lines with x==B
would be harvested to elasticsearchB
, and they each has a set of different processors.
Is it possible? Thanks.
No, unfortuanetly this is not possible. Yes, you can define multiple processors for the same input with each having their own condition as you've described (see https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html).
The problem is the configuration of the elasticsearch hosts in the filebeat.yml. Each event, regardless of the particular processor, would be distributed to all elasticsearch hosts since you can't define a specific elasticsearch host in the log inputs/processors themselves.
I think of two workarounds to meet your requirements:
install multiple filebeat instances/services each with a dedicated input and processor. In the particular filebeat.yml you then specify only the relevant host the data should get sent to.
setup Logstash as an intermediate component between filebeat and elasticsearch. Your filebeat would then send the events to a logstash pipeline. There you can implement a filter which checks for certain criteria of the events (e.g. tags you set through the processor) and then distribute them to the correct elasticsearch hosts. (Think of it as an if-else structure)
I would suggest that you go for the logstash-workaround since its easier to scale and fulfill future requirements.
I hope I could help you.
EDIT:
I modified my answer by more focussing on multiple processors for one input instead of multiple inputs. The idea of the workarounds is yet still valid.