Search code examples
elasticsearchelasticsearch-painless

Filter out null fields

I'm using map/reduce functionality with the painless language to do some aggregation.

In the map_script part I am trying to filter out all documents that have a certain field set to null but I am unable to do so.

I've tried doing containsKey() and != null but the expression evaluates to true for every document even though I've look at the underlying data and the vast majority of documents have this field set to null.

I can't explain this but it's behaving as this field's value for almost every document is treated as 0, not null, here when I'm aggregating it. But this is not the case because when I do query of this field, I see a bunch of nulls.

{
    "aggs": {
        "my_agg_name": {
            "scripted_metric": {
                "combine_script": "... combine script ...",
                "map_script": "if (doc.containsKey(\"my_field\") && doc.my_field.value != null) { ... } }",
                "init_script": "... init script ...",
                "reduce_script": "... reduce script ..."
            }
        }
    },
    "size": 0
}

Does anyone have an idea what's going on? How to filter out nulls in the map/reduce aggregation using painless?

Solution

  • I've figured it out. doc.my_field.value apparently transforms the null value into 0.

    What worked is using !doc.my_field.empty instead.