Search code examples
c#.netelasticsearchnest

NEST returns null instead of fields

I'm working with elastic for a long time, but I have never written a code that fetches some data. And now I'm in trouble.

I have an index where I want to retrieve some documents projected on some field. I could literally write it in SQL

SELECT myDocumentField
FROM myIndex

But for some reason I'm getting nulls instead of values.

I have 6 documents in my index. So I write following query:

var elasticServiceNumbers = await _elasticClient.SearchAsync<ElasticRequest>(
                                s => s.Query(Selector));

It works as expected and return these 6 values, except that all their fields are nulls

enter image description here

Okay, I'm trying to add fields as well:

var elasticServiceNumbers = await _elasticClient.SearchAsync<ElasticRequest>(
                                s => s.StoredFields(sf => sf.Fields(f => f.ServiceNumber))
                                      .Query(Selector));
var elasticServiceNumbers2 = await _elasticClient.SearchAsync<ElasticRequest>(
                                s => s.Source(sf => sf.Includes(fds => fds.Field(f => f.ServiceNumber)))
                                      .Query(Selector));

But still out of luck, and fields keep their null values.

Kibana shows that this fields exist on the index:

enter image description here

What could be wrong here?

Kibana query

{
  "version": true,
  "size": 500,
  "sort": [
    {
      "_score": {
        "order": "desc"
      }
    }
  ],
  "_source": {
    "excludes": []
  },
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "3h",
        "time_zone": "Asia/Baghdad",
        "min_doc_count": 1
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    "@timestamp",
    "fields.Date",
    "fields.DeserializedMessage.Message.Date",
    "fields.DeserializedMessage.Message.Periods.Begin",
    "fields.DeserializedMessage.Message.Periods.End",
    "fields.DeserializedMessage.Message.ResponseDate",
    "fields.Periods.Begin",
    "fields.Periods.End",
    "fields.ResponseDate"
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "gte": 1527973200000,
              "lte": 1528577999999,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "filter": [],
      "should": [],
      "must_not": []
    }
  },
  "highlight": {
    "pre_tags": [
      "@kibana-highlighted-field@"
    ],
    "post_tags": [
      "@/kibana-highlighted-field@"
    ],
    "fields": {
      "*": {}
    },
    "fragment_size": 2147483647
  }
}
Solution

  • It looks like the fields in question are doc value fields , in that they are searchable and aggregatable but are not stored i.e. the original _source document sent to Elasticsearch is not stored.

    To get the doc value fields with NEST

    var searchResponse = client.Search<ElasticRequest>(s => s
    .DocValueFields(f => f
        .Field(ff => ff.ServiceNumber.Suffix("keyword"))
    )
);

    This is using the keyword mapping of serviceNumber which looks to be a doc value field.