Search code examples
c#jsonelasticsearch.net-corenest

Indexing a large JSON using NEST


I'm trying to add a large JSON(around 22k lines) file to my ElasticSearch index but I am unable to do so since I am getting error response from ElasticSearch whatever I tried. I'm using ElasticSearch 7.6.2 and I'm using NEST to communicate with ElasticSearch.

I will add only the code related to the problem.

First I create connection settings like this:

_settings = new ConnectionSettings()
                               .DefaultMappingFor<string>(m => m  // I've put string here because that is the return type of File.ReadAllText() function
                               .IndexName("jsonindex")
                               );

Then I create my ElasticClient:

_elasticClient = new ElasticClient(_settings);

I've created my index succesfully like this:

_elasticClient.Indices.Create("jsonindex", c => c
                       .Settings(s => s
                           .NumberOfShards(1)));

When everything is setup, I do the following:

var json = File.ReadAllText("path\to\file.json"); //read the json file 
var indexResponse = _elasticClient.IndexDocument(jsonToSend); //try to index it, but this is where I get the error message

I get the following error:

Request failed to execute. Call: Status code 400 from: POST /jsonindex/_doc. ServerError: Type: mapper_parsing_exception Reason: "failed to parse" CausedBy: "Type: not_x_content_exception Reason: "Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes""

I've also tried calling the .Index() method and specify the index name as the second argument, but result is the same.


Solution

  • Looks like I was not going in the right direction while using the high level client, I accessed the low level client like this:

    _elasticClient.LowLevel.Index<StringResponse>("jsonindex", json);
    

    and it worked on the first try, it might help someone who was using high level client like me. (Note: I assume there is a way to achieve what I wanted to do using high level client, I just haven't figured it out yet)