How to keep the write operation incomplete until the atlas search has finished updating the new/updated documents

Question

I’m currently facing a challenge with my application where, after adding or updating an item, the changes are not immediately reflected when redirecting to the list page. I’ve noticed that introducing a delay of 1000ms seems to temporarily solve the issue, but it’s not a sustainable solution.

I suspect the problem lies in the synchronization between write operations and Atlas Search index updates. I’ve explored using the Atlas API and the MongoDB Node.js driver to check the indexing status, but so far, it hasn’t provided a reliable solution.

Have any of you encountered a similar issue or have suggestions on how to ensure that the write operation only completes when the Atlas Search index has finished updating? I’d greatly appreciate any insights, alternative approaches, or best practices you might have in dealing with this scenario.

Thank you in advance for your valuable input!

Answer 1

I encountered a similar issue before, and what I did was create a custom function to check if an index is updated. This function fetches inserted or updated documents using the $search query.

For instance, let's say we insert a new record. We want to make sure it's synced with the search index. To do this, we can use an aggregation query like the one below to check if the inserted document is present:

[{
    $search: {
        index: <<searchIndex>>,
        compound: {
            must: [
                {
                    in: {
                        path: "_id",
                        value: <<objectiveId>>
                    }
                }
            ]
        }
    }
}]

Execute the aggregation query until the desired result is achieved. For safety, set a threshold of a few seconds.

Use this function just after the write operation and ensure that the write operation API returns success only after the data is synced to the Atlas search index.

Similarly, for update operations, we can include the updated fields in the query to verify. And for delete operations, we can use a similar query but ensure that the document is not retrieved.