Search code examples
elasticsearchkibanaamazon-elasticsearch

Kibana including versioned documents in visualizations


I have a document with _id "123456", and when I do a GET in Elasticsearch for that ID in my index I can see that it is _version: 2 which makes sense because I updated it.

However in my Kibana visualizations it seems like it is picking up both versions of the same document when showing the results.

How do I exclude versioned documents from re-appearing in the visualization? For example, this record is showing up twice in my bar graph.

Please and thank you


Example GET response:

{
    "_index": "censored",
    "_type": "censored",
    "_id": "123456",
    "_version": 2,
    "found": true,
    "_source": {
        ... ommitted
    }
}

Also I am sure there is only one actual document with that ID because if I do a _search on the _id field I can see this:

{
    "took": 1,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 1,
        "max_score": 7.53924,
        "hits": [
            {
                "_index": "censored",
                "_type": "censored",
                "_id": "123456",
                "_score": 7.53924,
                "_source": {
                    ... ommitted
                }
            }

        ]
    }
}

EDIT: Things I've tried below

aggs": {
    "latest": {
      "terms": {
        "field": "_id"
      }
    }
}

and

"aggs": {
    "latest": {
      "max": {
        "field": "version"
      }
    }
}

Solution

  • So frankly this is just a workaround, if someone finds a better solution I will mark that as the answer instead. Anyway this is how I've been able to prevent multiple records with the same _id showing up in my visualizations on my dashboard:

    I just changed the "Y Axis - Count" on all the visualizations to being "Y Axis - Unique Count on field _id"

    Honestly it seems silly that I have to do this because I think different versions should just automatically be exempt from appearing in my saved searches & visualizations. I couldn't find any information about why this was happening. I even tried a _forcemerge to try and delete previous versions of records but it didn't do anything.

    Would be nice if someone found a real solution.