A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field

Question

when I run following query:

GET /annotations/_search
{
  "query": {
    "match_all": {}
  }
}

I receive following response:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "annotations",
        "_type" : "_doc",
        "_id" : "I9nlA",
        "_score" : 1.0,
        "_source" : {
          "preferences" : {
            "id" : 1,
            "annotation_id" : "I9nlA",
            "answer_timer" : 24
          }
         // other fields here
        }
      },
      {
        "_index" : "annotations",
        "_type" : "_doc",
        "_id" : "XIP6L",
        "_score" : 1.0,
        "_source" : {
          "id" : "XIP6L",
          "preferences" : {
            "id" : 2,
            "annotation_id" : "XIP6L",
            "answer_timer" : 5
          },
        // other fields here
        }
      }
    ]
  }
}

As you can see, the field Preferences has no null values. My problem is that when I add a scripted field based on the Preferences field, it raises an error. The query:

GET /annotations/_search
{
  "query": {
    "match_all": {}
  },
  "script_fields": {
    "can_answer": {
      "script": {
        "source": """
          if (doc['creator_id'].value == params['user_id']){
            return true;
          }
          else{
            String nowString = params['now'];
            ZonedDateTime now = ZonedDateTime.parse(nowString);
            ZonedDateTime created = doc['created'].value;
            ZonedDateTime createdPlusAnswerTimer = created.plusHours(
              doc['preferences.answer_timer'].value
            );
            Duration d = Duration.between(now, createdPlusAnswerTimer);
            return d.toHours() > 0;
            }
        """,
        "params": {
          "user_id": 1,
          "now": "2022-05-17T16:17:49.366Z"
        }
      }
    }
  }
}

The error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "runtime error",
        "script_stack" : [
          "org.elasticsearch.index.fielddata.ScriptDocValues.throwIfEmpty(ScriptDocValues.java:73)",
          "org.elasticsearch.index.fielddata.ScriptDocValues$Longs.get(ScriptDocValues.java:118)",
          "org.elasticsearch.index.fielddata.ScriptDocValues$Longs.getValue(ScriptDocValues.java:113)",
          "createdPlusAnswerTimer = created.plusHours(\n              doc['preferences.answer_timer'].value\n            );\n            Duration ",
          "                                                                                         ^---- HERE"
        ],
        "script" : " ...",
        "lang" : "painless",
        "position" : {
          "offset" : 398,
          "start" : 309,
          "end" : 441
        }
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "annotations",
        "node" : "_ljs4uLdR7eBjL8ioYLqAg",
        "reason" : {
          "type" : "script_exception",
          "reason" : "runtime error",
          "script_stack" : [
            "org.elasticsearch.index.fielddata.ScriptDocValues.throwIfEmpty(ScriptDocValues.java:73)",
            "org.elasticsearch.index.fielddata.ScriptDocValues$Longs.get(ScriptDocValues.java:118)",
            "org.elasticsearch.index.fielddata.ScriptDocValues$Longs.getValue(ScriptDocValues.java:113)",
            "createdPlusAnswerTimer = created.plusHours(\n              doc['preferences.answer_timer'].value\n            );\n            Duration ",
            "                                                                                         ^---- HERE"
          ],
          "script" : " ...",
          "lang" : "painless",
          "position" : {
            "offset" : 398,
            "start" : 309,
            "end" : 441
          },
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"
          }
        }
      }
    ]
  },
  "status" : 400
}

It said there is some problems with doc['preferences.answer_timer'].value, but I don't know why. This field has value in all documents.

What is the problem and how can I solve it? Thanks.

UPDATE: Mapping added:

 {
  "annotations" : {
    "mappings" : {
      "properties" : {
        
        // other fields
        
        "preferences" : {
          "type" : "nested",
          "properties" : {
            "annotation_id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "answer_timer" : {
              "type" : "long"
            },
            "id" : {
              "type" : "long"
            }
          }
        }
        
        // other fields
        
      }
    }
  }
}

NOTE when I use params._source.preferences.answer_timer, it works fine. But in the Elasticsearch documentation, it's been said that using _source.field is very slow, so I don't want to use it.

Answer 1

Since the preferences field is nested, we need to call the script in the context of the nested query. The parent document does not have a doc_value field - Thus giving an error. Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html

Solution 1: Change preferences type to object instead of nested.
Solution 2: use query in nested context.