Search code examples
elasticsearchlogstashkibana

How to do "where not exists" type filtering in Kibana/ELK?


I am using ELK to create dashboards from my log files. I have a log file with entries that contain an id value and a "success"/"failure" value, displaying whether an operation with a given id succeeded or failed. Each operation/id can fail an unlimited number of times and succeed at most once. In my Kibana dashboard I want to display the count of log entries with a "failure" value for each operation id, but I want to filter out cases where a "success" log entry for the id exists. i.e. I am only interested in operations that never succeeded. Any hints for tricks that would achieve this?


Solution

  • This is easy in Kibana 5 search bar. Just add a filter

    !(_exists_:"your_variable") 
    

    you can toggle the filter or write the inverse query as

    _exists_:"your_variable"
    

    In Kibana 4 and Kibana 3 you can use this query which is now deprecated

    _missing_:"your_variable"  
    

    NOTE: In Elasticsearch 7.x, Kibana now has a pull down to select KQL or Lucene style queries in the search bar. Be mindful that syntax such as _exists_:FIELD is a Lucene syntax and you need to set the pulldown accordingly.