Search code examples
c#elasticsearchasp.net-web-apikibananest

Search API for ElasticSearch not returning all the logs

I am wanting to get all the logs from elasticsearch from a asp.net core web api application using NEST. I created a controller named ESController to get the logs that are in elasticsearch. However, when I do so, on swagger it only displays the first log and not the entire list of logs. I even wrote to the console to see if I can view those logs there as well but it does not display. I do not have a model class for the logs. Is there something wrong I am doing to retrieve/extract the logs from elasticsearch in the .net application?

Console displays: "Valid NEST response built from a successful (200) low level call on POST: /elastic-search-app-logs%2A/_search?typed_keys=true"

enter image description here

swagger:

enter image description here

Correct me if I am wrong, but in the ElasticSearch CLI running the command below will display the logs from elasticsearch:

GET elastic-search-app-logs*/_search
{
  "query": {
    "match_all": {}
  }
}

So what I am trying to do is translate this code into .net with NEST.

enter image description here

The logs I am wanting to read/extract from elasticsearch into a .net application enter image description here

Any pointers/suggestions would be greatly appreciated!!

Solution

  • The _search API is limited in the number of results it will return, even with a match_all filter.

    The traditional way to get all results from Elasticsearch is scroll search. More recently the recommended approach is to use search_after instead (see the scroll search link for more info).

    Docs for the NEST client cover both search_after and scrolling.

    The _search API is limited to 10,000 results by default. While it's not recommended, you can change the limit with the index.max_result_window index setting.

    [edit: typo fix]