ElasticSearch: Replacing an entire batch of indexed items

Question

I would like to use ElasticSearch to index data items. These items are updated daily in batches (from an external source) - each batch replaces the batch from the day before. After the new batch is inserted, the old batch from yesterday can be deleted.

The data items are received in files, parsed by a scheduled job, and then should be persisted to ElasticSearch. I'd like to keep it simple with ElasticSearch only and not add another DB to the process.

How would you do it?

Answer 1

Create a daily index for your data , e.g data-2014-10-03 and use elasticsearch-curator to prune off indices older than 24 hours