Search code examples
loggingsplunksplunk-query

How to find duplicate log events in Splunk

I'm trying to query my Splunk logs to find duplicate data, but am unable to find the right query.

Example logs:

{"time":"2021-07-08 02:16:17.9232","level":"debug","message":"update","parameters":{"id":["1"], other params...}}
{"time":"2021-07-08 02:17:17.9232","level":"debug","message":"update","parameters":{"id":["1"], other params...}}

The duplicate log events would have the same id parameter, but a different timestamp.

Solution

  • Once you have the desired field extracted, finding duplicates is a matter of counting the instances of each value of that field and displaying the ones with a count greater than one.

    index=foo
| spath
| stats count by "parameters.id"
| where count > 1