Search code examples
cadence-workflow

Should Uber Cadence activities be part of service implementations?

I have a question about “best practice” for implementing activities in Cadence. When the activities of a workflow span different services, are activities typically implemented as part of the services themselves or is it more common to keep the activities separate and rely on service API:s to interact with the services?

Solution

  • The reasons to embed activities directly into a separate service:

    • Long running operations: There is no standard and clean way to implement a long running operation in an RPC service. If an activity can take minutes or more to run it usually is expected to heartbeat to ensure timely timeout. The Cadence client library directly supports heartbeating.
    • Flow control: The Cadence worker configuration specifies the maximum consumption rate and the maximum number of activities being processed simultaneously. The Cadence worker is essentially a queue consumer that receives new activity tasks only when it has capacity to work on them according to the configuration. When an activity calls into a remote service this flow control is lost. When overloaded a remote service can only fail a request. Cadence activity does support exponential retry in this case, but relying on failures and retries for flow control is clearly an inferior solution.

    There are case when embedding is not possible:

    • Calling external services: Frequently a workflow has to depend on existing services that cannot embed Cadence activities. In this case an activity calling an external service is the only option. For flow control make sure to specify an exponential retry policy (including the list of non-retryable errors) when executing the activity. For long running operations model the invocation as two activities. The first one invokes the start whatever API and the second polls for the result in a loop inside the activity function. The PollForResult activity should heartbeat back to Cadence service to make sure that it is retried if the worker that hosts it goes down.
    • Unsupported programming language: If you have to implement an activity in a language that still doesn't have a corresponding Cadence client side library then exposing this function as a service frequently is the simplest option.