Search code examples
architecturemessage-queuecadence-workflowtemporal-workflow

Temporal / Cadence orchestration concept

In current application, we have three services:

  • invoice
  • bank transfer : calls external API, which might takes a minute or so depends on queue
  • ledger : internal microservice that create debit-credit ledger

To communicate, we have an orchestration saga. The flow is basically using invoice as the orchestrator:

  1. invoice service publish message to rabbitmq, asking payment for invoice X
  2. bank transfer service listen message, get invoice X
  3. bank transfer service process the transfer (calls bank's API). When payment success, publishing message to rabbitmq that 'Invoice X paid'
  4. invoice service listen the message 'Invoice X paid'
  5. Invoice service publish message 'Create ledger for invoice X' and publish to rabbitmq
  6. ledger service listen from rabbitmq for message 'Create ledger for invoice X'
  7. ledger service create appropriate ledger journal debit / credit, then publish 'ledger created for invoice X'
  8. Invoice service listen for message 'ledger created for invoice X', then finalize (close) the transaction

Question 1 If I'm using Temporal or cadence (just looking for it), they doesn't rely on pub-sub pattern like above. So how do I implement it?

I'm thinking (CMIIW) :

  • create InvoiceActivity
  • create BankTransferActivity
  • create LedgerActivity
  • create InvoicePaymentWorkflow

All of those Workflow and Activity are part of invoice service app. But instead of relying on pub-sub, the bank transfer service and ledger service now provides API to process transfer (in term of bank transfer, it acually a proxy API to external bank calll), and ledger service provides API to create debit/credit journal. CMIIW, in this case, we no longer need creating listener (or cadence / temporal activity) on bank transfer service and ledger service, we need to provides API.

OR I'm thinking it wrong? There should be some activity on bank transfer service and ledger service? But if so, how can the invoice service trigger and arrange the workflow?

Question 2 But now the call is asynchronous. The journal ledger creation actually has some validation, and heavy load at times, so depends on message on queues, it can take up to 5 minutes from the message goes to rabbitmq until journal actually created. On API call, this will be a timeout.

Question 3 And what about the race condition? Some journals to be created in sequence. Using rabbitmq, we can achieve this with certain technique (single consumer on queue, something like kafka topic).

Question 4 Also, how to handle the compensating transaction? If ledger failed, we must do something. In our case, we must notify somebody on accounting, since bank transfer already processed and cannot reversed.

Question 5 For some subsidiariy companies, when ledger failed, we still has control, so we can compensate / withdraw the invoice amount, and return the amount back to parent company. Suppose we have a listener for this on the bank transfer service, how can we trigger the compensation API?

In cadence / Temporal, is this suitable usecase? If so, how to handle those issues above?

Thanks

Solution

  • TLDR; This use case is the perfect fit for Temporal as it greatly simplifies your code and operations.

    Question 1

    I would recommend hosting activities in their correspondent services. So InvoiceActivity should be hosted by the invoice service and BankTransferActivity be hosted by the bank transfer service. This eliminates the need for creating synchronous APIs and ensures flow control between workflow and activity implementation. See this post that explains this in more detail.

    The invoice service app would host the workflow code only in this case.

    Question 2

    Hosting activity in the corresponding service solves this issue. Temporal supports activities of unlimited duration. Note that heartbeating is recommended for long-running activities to ensure timely failure detection.

    Question 3

    The exact solution depends on the exact requirements. In the majority of cases, the journal entries that require specific sequencing should be orchestrated from a single workflow.

    Question 4

    Temporal by ensuring that the workflow code eventually completes makes it trivial to support compensating actions. Here is a SAGA example from Java SDK Samples repository.

    Question 5

    Just make this compensation logic part of the same workflow.