Search code examples
pythonelasticsearchopensearchamazon-opensearch

AWS bulk indexing using gives 'illegal_argument_exception', 'explicit index in bulk is not allowed')

While I am trying to bulk index on AWS Opensearch Service (ElasticSearch V 10.1) using opensearch-py, I am getting below error

RequestError: RequestError(400, 'illegal_argument_exception', 'explicit index in bulk is not allowed')

from opensearchpy.helpers import bulk
bulk(client, format_embeddings_for_es_indexing(embd_data, titles_, _INDEX_))

format_embeddings_for_es_indexing() function yeilds

{
'_index': 'test_v1',
'_id': '208387',
'_source': {
    'article_id': '208387',
    'title': 'Battery and Performance',
    'title_vector': [ 1.77665558e-02,  1.95874255e-02,.....],
    ......
    }
}

I am able to index documents one by one using `open search.index()'

failed = {}
for document in format_embeddings_for_es_indexing(embd_data, titles_, _INDEX_):
    res = client.index(
        **document,
        refresh = True
    )
    if res['_shards']['failed'] > 0:
        failed[document["body"]["article_id"]] = res['_shards']

# document body for open search index
{
'index': 'test_v1',
'id': '208387',
'body': {
    'article_id': '208387',
    'title': 'Battery and Performance',
    'title_vector': [ 1.77665558e-02,  1.95874255e-02,.....],
    ......
    }
}

please help

Solution

  • This may have something to do with what is documented here: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html#ac-advanced

    Please make sure that the value of rest.action.multi.allow_explicit_index in the Advanced cluster settings is true