I discovered Filebeat a couple days ago. I have it sending data to Kafka directly if I hard code the topic name in filebeat.yml. But I can't seem to figure out how to dynamically compute the topic name based on suricata event type. I've enabled the filebeat suricata module, and tried a number of things in the filebeat.yml topic value, like:
topic: 'suricata-%{[fields.suricata.eve.event_type]}'
But I always get this error in the log:
2020-01-14T23:44:49.550Z INFO kafka/log.go:53 kafka message: Initializing new client
2020-01-14T23:44:49.551Z INFO kafka/log.go:53 kafka message: Successfully initialized new client
2020-01-14T23:44:49.551Z INFO pipeline/output.go:105 Connection to kafka(somehost:9092) established
2020-01-14T23:44:49.551Z ERROR kafka/client.go:144 Dropping event: no topic could be selected
2020-01-14T23:44:49.551Z ERROR kafka/client.go:144 Dropping event: no topic could be selected
2020-01-14T23:44:49.551Z ERROR kafka/client.go:144 Dropping event: no topic could be selected
How do you do it? Any sample filebeat.yml files around that route to different topics based on suricata event-type?
In case there is another person on earth interested in this, I got an answer that works here:
https://discuss.elastic.co/t/suricata-logs-to-filebeat-to-kafka-topics-by-event-type/215179
topic: 'suricata-%{[suricata.eve.event_type]}'