In the backend of my app, there is a route that first process payment and then writes to a collection in my MongoDB. So:
await call_payment_vendor_api();
// system fails here
await write_to_collection();
Now there is an inconsistency in the system. How do I recover from that?
Transactions across systems are often a bit tricky and there are multiple approaches to handle this. There is no silver bullet, though, which is why many so-called microservice systems end up as distributed monoliths.
The most straightforward way if you're in a sync-context (I.e. need to answer a web request immediately ): try/catch the exception from MongoDB, then refund the payment if it fails.
For a better user experience, I'd try putting the writing into a background job queue, which processes the pending update and maybe retries a couple of times before giving up and refunding. Or maybe escalates to a technical support who can take a look and maybe fix things through a back-office admin UI. But again, in the context of a web request, you might have to poll the job status to update the website or re-design the flow altogether.
Another possibility: In MongoDB, create a transaction, first write your data, then call out to the payment provider. If they confirm, you'll only have to commit the transaction which (usually) is much less likely to fail.