Raise an alert in prometheus when logstash's elasticsearch output plugin log an error

Question

I am using logtash 6.6.2 to send logs to elasticsearch. When logstash is unable to send a log record to elasticsearch , elasticsearch output plugin log an error in the logstash container. I am wondering if there is a way to raise an alert in prometheus with some metrics provide by logstash to be aknowledge to that error. I looked at the metrics provide on the metrics endpoint of logstash but none of the provide metrics suits.
PS: I used a kubernetes cluster to deploy my applications

Answer 1

The metrics provided by logstash in the API are not enough for a consistent alerting.

The alternatives can be:

• use mtail for parsing the logs and increment a counter whenever you get this specific error.
• use a blackbox exporter for checking the availability of ES
• find a way to detect a flatline in your log ingestion (logstash reports events in output which are inconsistent with ES input) - this depends on your setup