Handling Multi-State Systems Avoiding Inconsistent States

Question

Here's my use-case:

I have a service (service 1) that needs to update the state of some entity in multiple external services (service 2, service 3), on getting a request to do so. I also have to update the state local to me (service 1). I am trying to handle the failure scenarios in this case:

1. I successfully update the states in external service 2 and 3, and I fail to update the state in service 4.
2. I successfully updated the states in external service 2 and 3, and I died.
3. I successfully update the states in all the external services, but I fail to persist the state in my local system.

Say that the service 2 does a refund of products, and the service 3 marks the invoice as paid. Now, if I successfully issue the refund (service 2) but fail to mark the invoice as paid (service 3), I'll have a serious business implication during financial reconciliation.

In all of these cases, is there a proper way to handle the failing scenarios?

Answer 1

The proper way is using distributed transactions and 2-phase commit, e.g. applying the XA protocol.