Search code examples
elasticsearchlogstashdeduplication

How do I keep a count of deduplicated messages from Logstash in ElasticSearch?


I see from this question that document_id can easily be used in Logstash to replace a duplicate record in ElasticSearch. How would I add/increment a count value for e.g. repeating syslog messages? Instead of just replacing the record I want to increment the count so I know how many times it has repeated.


Solution

  • Depending on what you are using to view the data, it might just be as simple as looking at the _version field of the documents. ES will automatically update that value when something changes for the document. Kibana doesn't show the _version field (https://github.com/elasticsearch/kibana/issues/1234), but it's there.