How to safely recreate an index with .NET client Elasticsearch

Question

I am using Elasticsearch NEST 6.x, and in my indexer, I would like to delete the existing index and recreate a new one as my data gets updated. This is what I currently have:

this.elasticClient.DeleteIndex(indexName, x => x.RequestConfiguration(y => y.AllowedStatusCodes(404)));
this.elasticClient.CreateIndex(indexName, x => x.Settings(s => s.NumberOfShards(1)).Mappings(ms => ms.Map<T>(m => m.AutoMap())));
this.elasticClient.Bulk(b => b.Index(indexName).IndexMany(entities));

The problem with this approach is that if delete is successful but create fails, that index won't be available until it's recreated. Further, even in the best case where it succeeds, in the meanwhile the index is being recreated, all the searches to that index will fail since it does not exist. Is there a better way?

Edit:

I think it's worth mentioning why I recreate the index as @leandrojmp asked:

My data is not additive, i.e. over time, some indexed entities need to be added, some need to be updated, and some need to be removed. On the other hand, the amount of data I have is small enough that this approach sounds feasible.

However, I'm new to elastic search and would love to know if there is a better way.

Answer 1

You may want to consider using index aliases and

1. create versioned indices e.g. my_index_v1
2. create an alias that points to a single index e.g. my_index alias that points to my_index_v1
3. Use the my_index alias for bulk indexing and search

Then when wishing to delete and create the index,

1. create a new index my_index_v2
2. send a request to the alias API to remove the my_index alias from my_index_v1 and add to my_index_v2
3. delete my_index_v1

In doing so, any applications and operations can use the index alias. With this approach, it is possible that data you intended to be indexed into my_index_v2 is actually indexed into my_index_v1, because the bulk index request is processed before my_index_v2 is created and the index alias is swapped over. If this is an issue, then you could make my_index_v1 not writable when creating my_index_v2 and swapping the alias over, and handle the failure to bulk index in your application, potentially queueing those documents that fail to be indexed and trying again; you might check for a specific failure response from bulk indexing, and check the index metadata to see if the index is writable.