Search code examples
elasticsearchelasticsearch-7

Bulk Insert object in Elasticsearch

I am trying create an index and then do a bulk insert using RestHighLevelClient to my ES (the code is in Kotlin).

The bulk insert code is :

private fun insertEntity(entityList: List<Person>, indexName: String) {
    var count = 0
    val bulkRequest = BulkRequest()

    entityList.forEach {
        bulkRequest.add(IndexRequest(indexName).source(it,XContentType.JSON))
        count++

        if (count == batchSize) {
            performBulkInsert(bulkRequest)
        }
    }
}

When executing this, I am getting an exception saying : Limit of 1000 fields is crossed.

On analysing my code, I feel the implementation is wrong, because :

bulkRequest.add(IndexRequest(indexName).source(it,XContentType.JSON))

source takes a String type but I am passing the Person (it)object itself. So I believe that is causing some issue related to 1000 fields based on my mapping or something.

Not sure if my assumption is correct. If yes, how can I achieve the bulk insert then ?

EDIT

Index creation:

private fun createIndex(indexName: String) {
    val request = CreateIndexRequest(indexName)

    val settings = FileUtils.readFileToString(
        ResourceUtils.getFile(
            ResourceUtils.CLASSPATH_URL_PREFIX + "settings/settings.json"), "UTF-8")

    val mappings = FileUtils.readFileToString(
        ResourceUtils.getFile(
            ResourceUtils.CLASSPATH_URL_PREFIX + "mappings/personMapping.json"), "UTF-8")

    request.settings(Settings
        .builder()
        .loadFromSource(settings, XContentType.JSON))
        .source(mappings, XContentType.JSON)
    restHighLevelClient.indices().create(request, RequestOptions.DEFAULT)
    
}

Mapping.json Please note original has 16 fields.

{
  "properties": {
    "accessible": {
      "type": "boolean"
    },
    "person_id": {
      "type": "long"
    },
    "person_name": {
      "type": "string",
      "analyzer": "lower_keyword"
    }
}
}

Thanks.

Solution

  • Looks like you are using the dynamic mapping and due to some mistake when you index a document it ends up creating new fields in your index which crossed the 1000 fields limit.

    Please see if you can use the static mapping or debug the code which prepares the document and compare it with your mapping to see if its creating new fields.

    Please refer this SO answer to increase the limit if its legitimate or use static mapping or debug the code to figure out why you are adding new fields to elasticsearch index.