Search code examples
amazon-web-servicesamazon-swf

Routing an activity task to a specific worker in the SWF fleet

I have a fleet of multiple worker hosts polling for the following tasks of my SWF:

  1. Activity 1: Perform some business logic to create a large file.
  2. Activity 2: Wait for some time (a human approval, timer, etc.)
  3. Activity 3: Transmit the file using some protocol (governed by input parameters of the SWF).
  4. Activity 4: Clean-up the local-generated file.

The file generated in Step-1 needs to be used again in Step-3, and then eventually discarded at the end of the workflow.

The system would work fine if there is only 1 host polling for all tasks. However, when I have multiple workers, I cannot seem to ensure that task-1 and task-3 would end up on the same host.

I would like to avoid doing the following:

  • Uploading the file to a central repository (say S3) on step-1 and download it in step-3; or
  • Having a single activity for the task-1 and task-3.

I have the following questions:

  • Is it possible to control that subsequent activities be run on the same host as opposed to going to any random host in my fleet?
  • What are specific guidelines/best practices on re-using resources generated in different activities in a workflow?
Solution
    • Is it possible to control that subsequent activities be run on the
      same host as opposed to going to any random host in my fleet?

    Yes, absolutely. The basic idea is that SWF task lists (queues used to deliver activity tasks) are dynamic. So each host can have its own task list and workflow can specify specific task list name when calling an activity. See fileprocessing sample which executes download activity on any host from the pool, then converts the file and uploads the result on the same host as the first one.

    • List item What are specific guidelines/best practices on re-using resources generated in different activities in a workflow?

    The approach of caching result in the worker process memory or on the local disk is considered the best practice. Sometimes using external data store and getting it each times also makes sense.