Elasticsearch exception : mappping depth in index has been exceeded

Question

I have setup an Elasticsearch cluster on Bonsai. I am using elasticsearch-rest-high-level-client library to read Twitter-tweets stored in Kafka and push them to the Elasticsearch index.

I getting below exception : Exception in thread "main" ElasticsearchStatusException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of mapping depth [20] in index [twitter] has been exceeded due to object field [THIS CONTAINS ALL OF THE JSON MESSAGE RETRIEVED FROM KAFKA]

It seems my code is trying to put all the message as a single field. What could be going wrong?

IndexRequest indexRequest = new IndexRequest("twitter").source(jsonMessage, XContentType.JSON); IndexResponse indexResponse = restClient.index(indexRequest, RequestOptions.DEFAULT);

Answer 1

This error is about the depth of the Json you are trying to index. Default Mapping depth of 20 denotes that the Json you are trying to index can have fields going maximum of 20 levels deep. For example

For example in below json, shippingAddress is 3 levels deep

{
"order": {
    "orderNo": "test_order",
    "orderDate": "2020-01-01",
    "customer": {
        "customerName": "John Doe",
        "customerPhone": "555-555-5555",
        "shippingAddress": {
            "addressLine1": "test",
            "addressLine2": "test",
            "city": "test",
            "state": "test",
            "country": "test"
        }
    }
}

}

Try to optimize your document if possible. If not, the setting can be updated if really required,

index.mapping.depth.limit - The maximum depth for a field, which is measured as the number of inner objects. For instance, if all fields are defined at the root object level, then the depth is 1. If there is one object mapping, then the depth is 2, etc. The default is 20.

Check the documentation

https://www.elastic.co/guide/en/elasticsearch/reference/7.6/mapping.html#mapping-limit-settings