Search code examples
loggingalertmonitoringdatadog

Unique filter in datadog event monitor

I am looking for a unique filter in the Datadog event monitor. something like below.

enter image description here

Scenario:

I have multiple microservices running in Kubernetes. Each of the services will produce a log message in the format foreground-process thread-<ID> is in waiting for state. For each thread-<ID> multiple log messages would be produced. I am using the pipeline feature with grok parser to fetch threadType i.e foreground-process and thread-<ID> .i.e is thread-11. I need to create a monitor and alert when more than 5 unique threads are blocked per service. I can achieve this by creating a separate monitor for each service but then I need to create around 120 monitors. So I am looking to see if there is unique filter in datadog or any other mechanism to achieve this?

Sample logs:

foreground-process thread-2 is in waiting for state
foreground-process thread-11 is in waiting for state
foreground-process thread-2 is in waiting for state
foreground-process thread-9 is in waiting for state
foreground-process thread-2 is in waiting for state
Solution

  • To count the unique number of things, change that * in the yellow Count box to the facet you want to count unique values for.

    count unique in monitor