Search code examples
elasticsearchindexingelasticsearch-bulk-api

Get all documents from an index of an elasticsearch cluster and index it in another elasticsearch cluster

My goal here is to get all documents from an index of an ES cluster and insert them in another ES cluster keeping the same metadata.

I had a look at mget API to retrieve data and Bulk API to insert it but this Bulk API needs a special structure:

action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n

So my idea is to retrieve my data of EScluster1 in a file and rearranged it to meet the structure of Bulk API and index it to EScluster2.

Do you see a better and/or faster way to proceed?

Solution

  • elasticdump does this. If you want to do this manually, you'll want to query using scroll and then bulk index what comes out of that. Not too hard to script together. With elastic dump you can pump the data around without writing to a file. However, it is kind of limited when you have e.g. parent/child relations in your index.