How to migrate data from elasticsearch 5.6 to elasticsearch 8.3
I have an elastic search cluster running 5.6. I plan to upgrade my cluster but i plan to do it by basically running a ES cluster 8.3 running in parallel and then moving data over to it.
The preferred way i think is to do snapshot and restore https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html
But I am confused about what exactly Snapshot version compatibility mean : 
In the above does it mean that if i take a snapshot in elasticsearch 5.6, I cannot restore directly in version 8.3 ?? (which mean I have to first move to 6.x then to 7.x and finally to 8.x ) ??
The below index compatibility matrix, however says that a version in 5.x will work in 8.x ?
Am i missing something ? or can someone help me elaborate this?
So, the underlying problem is that data written in Lucene version N, can only be read with N+1. For Elasticsearch 5 to 8 the Lucene version was always 1 greater than the ES version (so 6 to 9).
That means, both for an upgrade or a restored snapshot: If your data was written with 5.x, you can only read / restore it with 6.x. For 7.x or 8.x you'll need to reindex the data. I would do a remote reindex straight from 5.x to 8.latest if possible: https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade-remote.html
There are some small caveats but they will probably not apply to you:
- This doesn't apply to source only snapshots, but those always need a reindex, so that's not going to add any benefit for you.
- 8.3 added a feature to still read snapshots from 5.0 on but it is slower, doesn't support all features, and it is a commercial feature (platinum license if I'm not mistaken).
Depending on what kind of data it is: If it's aging out (like logs or metrics), maybe you don't have to migrate it to the new cluster?
- Constrain a Specman list so it doesn't have identical values in consecutive elements
- How to type cast a list of uint to a list of vr_ahb_data in Specman?
- Static fields/methods in e
- What is the difference between deep_copy and gen keeping in Specman?
- Specman UVM: What is the difference between write_reg { .field == 2;}; and write_reg_fields?
- Does Specman support optional parameters to a method?
- Specman e: When colon equal sign ":=" should be used?
- Specman e: Is there a way to know how many values there is in an enumerated type?
- Specman e error: No match for file when using "for each line in file"
- How to run e file one by one? Not in parallel test
- Specman/e constraint (for each in) iteration
- Specman e: How driver's items queue can be locked from a sequence?
- Specman e: How to print variable's address?
- Specman e: "keep type .. is a" fails to refine the type of a field
- Specman soft select on variable, decimal vs. hexadecimal values
- Specman e: How to print a pointer to a struct?
- Specman e: Is there a way to print unit instance name?
- Specman e: simultaneous events error
- Specman e coverage: ignored values appear in the coverage statistics
- Specman e: How a sequence should be started when gen_and_start_main constrained to FALSE?
- Specman/e list of lists (multidimensional array)
- Does Specman e have struct constructor?
- Specman e: How the predefined sequence.item should be used?
- Specman e: define-as-computed macro error
- Specman e UVM: Why to inherit from uvm_* units?
- Specman e: A sequence drives its BFM also its MAIN was not defined in a test
- e HVL (IEEE 1647): expect expression fails unexpectedly
- Specman e subtyping: How to refer to FALSE value of conditional field in when/extend subtyping?
- e HVL (IEEE 1647): How to set 'X' value?
- Difference between declaring an event that is sensitive to a simple_port value and event_port