Search code examples
elasticsearchelasticsearch-bulk-api

ElasticSearch, use of Wildcard on index names while using bulk api

I'm using RabbitMQ River Plugin for Elasticsearch to Insert/Delete data to our ElasticSearch indexes. River plugin only supports Bulk API. Our current data is using multiples indexes, an index by date, something like “indexName_yyyymmdd”

When deleting a given ID, we don’t know which index the ID could be locate. We thought we will be able to use wildcard format, like “indexName*” in the same way you can do in searches.

We have tried, different combinations, but it looks that either we are not using the correct syntax or wildcards are definitely not supported ???

Examples we have tried:
// using default index in the url …
POST /mybestfares_gb*/bestfares_data/_bulk
{ "delete": {"_id": " AUry2F0Mu2oVvm5kYHaK " }}

// index in the meta section … 
POST /_bulk
{ "delete": {"_index":"mybestfares_gb*", "_type":"bestfares_data","_id": "AUry2F0Mu2oVvm5kYHaP" }}

In both cases we get the following error: "error": "InvalidIndexNameException[[mybestfares_gb*] Invalid index name [mybestfares_gb*], must not contain the following characters [\\, /, *, ?, \", <, >, |, , ,]]"

In the ElasticSeach documentation this is being mentioned: “Single index APIs such as the Document APIs and the single-index alias APIs do not support multiple indices.”

And Bulk API is classified as a Multi-Document API, so I'm hopping is just a matter of me no using the correct syntax?

Solution

  • Unfortunately that's not going to work. If you read the documentation closely:

    Single index APIs such as the Document APIs and the single-index alias APIs do not support multiple indices.

    and then follow the links to the list of Document API's:

    Document APIs

    This section describes the following CRUD APIs:

    Single document APIs

    • Index API
    • Get API
    • Delete API
    • Update API

    Multi-document APIs

    • Multi Get API
    • Bulk API
    • Bulk UDP API
    • Delete By Query API

    you will see that the Bulk API and the Delete and Delete By Query API are all classified as Document API's which cannot use a wildcard to cover multiple indexes:

    All CRUD APIs are single-index APIs. The index parameter accepts a single index name, or an alias which points to a single index.

    http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs.html