Update Elastic Search document immediately after put

Question

I would like to get an advice on how to deal with document visibility if I need to update a document soon after I add it.

My code writes many documents into a rolling index and a few min later update them with updateByQuery with additional info (I use updateByQuery since the rolling index may roll between calls). In some cases the update is too fast and the document is not visible so fail to update it. What should be the best practice in this case?

1. Keep retrying updates until the document is visible? I'm not sure how long it can take since visibility probably depends on the server CPU load and the size of shards.
2. Use ?wait_for flag when adding the first document (In this case I'm concerted from performance impact since most of the updates are done when the document is already visible so there is no need to delay all to first put just for few cases.
3. Alway overwrite the full document with the same id so the last update will be the latest ( but since I'm using rolling index I'm afraid that it can create two documents with the same id and will not overwrite each other.

What do you think will be the best practice in this case?

Answer 1

unfortunately, there's no magic solution here due to your use case

the best bet is to use option 2 - ?wait_for. there is a performance impact, but then you know for sure that you have consistency for your next operations

alternatively, maybe there's a different architecture you can use. but that would require understanding more information on your use case, and probably put into another question