NEST Api SearchAfter return null in NEST but works in Kibana

Question

We are using elastic search just for document search in our application so we don't have any one expert in it. I was able to use TermQuery, SimpleQueryStringQuery and MatchPhraseQuery successfully. But I found out in documentation that using From & Size for pagination is not good for production and Search After is recommended.

But my implementation return null. It is confusing for me what should be in <Project> parameter as shown in Nest API Object Initializer Syntax in docs here.

My code looks like this:

var request = new SearchRequest<ElasticSearchJsonObject._Source>
 {
    //Sort = new List<ISort>
    //{
    //    new SortField { Field = Field<ElasticSearchJsonObject>(p=>)}
    //},
    SearchAfter = new List<object> {

    },                    
    Size = 20,
    Query = query
  };                               

Reality is I don't understand this. Over here ElasticSearchJsonObject._Source is the class to map returned results.

My documents are simple text documents and I only want documents sorted according to score so document Id is not relevant.

There was already a question like this on SO but I can't find it somehow.

---

Update

After looking at answer I updated my code and though query obtained does work. It return result in kibana but not in NEST.

This is the new updated code:

var request = new SearchRequest<ElasticSearchJsonObject.Rootobject>
            {
                Sort = new List<ISort>
                {
                    new SortField { Field = "_id", Order = SortOrder.Descending}
                },
                SearchAfter = new List<object> {
                   "0fc3ccb625f5d95b973ce1462b9f7"
                },                    
                Size = 1,
                Query = query
            };

Over here I am using size=1 just for test as well as hard code _id value in SearchAfter.

The query generated by NEST is:

{
  "size": 1,
  "sort": [
    {
      "_id": {
        "order": "desc"
      }
    }
  ],
  "search_after": [
    "0fc3ccb625f5d95b973ce1462b9f7"
  ],
  "query": {
    "match": {
      "content": {
        "query": "lahore",
        "fuzziness": "AUTO",
        "prefix_length": 3,
        "max_expansions": 10
      }
    }
  }
}

The response from the ES does say successful but no results are returned.

• Results do return in Kibana
• Query status is successful
• But...
• Total returned is 0 in NEST
• Sort value is null in kibana I used TrackScores = true to solve this issue

Here is the debug information:

Valid NEST response built from a successful low level call on POST: /extract/_source/_search?typed_keys=true
# Audit trail of this API call:
 - [1] HealthyResponse: Node: http://localhost:9200/ Took: 00:00:00.1002662
# Request:
{"size":1,"sort":[{"_id":{"order":"desc"}}],"search_after":["0fc3ccb625f5d95b973ce1462b9f7"],"query":{"match":{"content":{"query":"lahore","fuzziness":"AUTO","prefix_length":3,"max_expansions":10}}}}
# Response:
{"took":3,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}

So please tell me where I am wrong and what can be the problem and how to solve it.

---

Update 2:

Code in Controller:

Connection String:

var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node);
settings.DisableDirectStreaming();
settings.DefaultIndex("extract");
var client = new ElasticClient(settings);

Query:

var query = (dynamic)null;
query = new MatchQuery
 {
    Field = "content",
    Query = content,
    Fuzziness = Fuzziness.Auto,
    PrefixLength = 3,
    MaxExpansions = 10
   };

Query Builder

var request = new SearchRequest<ElasticSearchJsonObject.Rootobject>
            {
                Sort = new List<ISort>
                {
                    new SortField { Field = "_id", Order = SortOrder.Descending}
                },
                SearchAfter = new List<object> {
                   documentid //sent as parameter
                },                    
                Size = 1, //for testing 1 other wise 10
                TrackScores = true,
                Query = query
            };

JSON Query I use this code to get query I posted above. This query is then passed to kibana with GET <my index name>/_Search and there it works

var stream = new System.IO.MemoryStream();
client.SourceSerializer.Serialize(request, stream);
var jsonQuery = System.Text.Encoding.UTF8.GetString(stream.ToArray());

ES Response

string responseJson = "";
                ElasticSearchJsonObject.Rootobject response = new ElasticSearchJsonObject.Rootobject();
                var res = client.Search<object>(request);
                if (res.ApiCall.ResponseBodyInBytes != null)
                {
                    responseJson = System.Text.Encoding.UTF8.GetString(res.ApiCall.ResponseBodyInBytes);
                    try
                    {
                        response = JsonConvert.DeserializeObject<ElasticSearchJsonObject.Rootobject>(responseJson);
                    }
                    catch (Exception)
                    {
                        var model1 = new LoginSignUpViewModel();
                        return PartialView("_NoResultPage", model1);
                    }
                }

This is where things go wrong. Above debug information was captured from response

ElasticSearchJsonObject

Some how I think problem might be here somewhere? The class is generated by taking response from NEST in Search request.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;

namespace ESAPI
{
    public class ElasticSearchJsonObject
    {
        public class Rootobject
        {
            public int took { get; set; }
            public bool timed_out { get; set; }
            public _Shards _shards { get; set; }
            public Hits hits { get; set; }
        }

        public class _Shards
        {
            public int total { get; set; }
            public int successful { get; set; }
            public int skipped { get; set; }
            public int failed { get; set; }
        }

        public class Hits
        {
            public int total { get; set; }
            public float max_score { get; set; }
            public Hit[] hits { get; set; }
        }

        public class Hit
        {
            public string _index { get; set; }
            public string _type { get; set; }
            public string _id { get; set; }
            public float _score { get; set; }
            public _Source _source { get; set; }
        }

        public class _Source
        {
            public string content { get; set; }
            public Meta meta { get; set; }
            public File file { get; set; }
            public Path path { get; set; }
        }

        public class Meta
        {
            public string title { get; set; }
            public Raw raw { get; set; }
        }

        public class Raw
        {
            public string XParsedBy { get; set; }
            public string Originator { get; set; }
            public string dctitle { get; set; }
            public string ContentEncoding { get; set; }
            public string ContentTypeHint { get; set; }
            public string resourceName { get; set; }
            public string ProgId { get; set; }
            public string title { get; set; }
            public string ContentType { get; set; }
            public string Generator { get; set; }
        }

        public class File
        {
            public string extension { get; set; }
            public string content_type { get; set; }
            public DateTime last_modified { get; set; }
            public DateTime indexing_date { get; set; }
            public int filesize { get; set; }
            public string filename { get; set; }
            public string url { get; set; }
        }

        public class Path
        {
            public string root { get; set; }
            public string _virtual { get; set; }
            public string real { get; set; }
        }
    }
}

I am sure this can be used to get response.

Please note that in case of simple search this code works:

so for this query below my code is working:

var request = new SearchRequest
                {
                    From = 0,
                    Size = 20,
                    Query = query
                };

Answer 1

Using from/size is not recommended for deep pagination because of the amount of documents that need to be fetched from all shards for a deep page, only to be discarded when finally returning an overall ordered result set. This operation is inherent to the distributed nature of Elasticsearch, and is common to many distributed systems in relation to deep pagination.

With search_after, you can paginate forward through documents in a stateless fashion and it requires

• the documents returned from the first search response are sorted (documents are sorted by _score by default)
• passing the values for the sort fields of the last document in the hits from one search request as the values for "search_after": [] for the next request.

In the Search After Usage documentation, a search request is made with sort on NumberOfCommits descending, then by Name descending. The values to use for each of these sort fields are passed in SearchAfter(...) and are the values of Project.First.NumberOfCommits and Project.First.Name properties, respectively. This tells Elasticsearch to return documents that have values for the sort fields that correspond to the sort constraints for each field, and relate to the values supplied in the request. For example, sort descending on NumberOfCommits with a supplied value of 775 means that Elasticsearch should only consider documents with a value less than 775 (and to do this for all sort fields and supplied values).

If you ever need to dig further into any NEST documentation, click the "EDIT" link on the page:

which will take you to the github repository of the documentation, with the original asciidoc markdown for the page:

Within that page will be a link back to the original NEST source code from which the asciidoc was generated. In this case, the original file is SearchAfterUsageTests.cs in the 6.x branch