Is it possible to add a slight delay to parallel luigi.Tasks?

Question

I have a number of luigi.Tasks in a workflow that have no dependencies. However, each of these task sends a command to a server that can get overwhelmed if I do not slightly stagger sending the commands.(i.e. put a 5 second delay between sending each command) Any way to handle this with an argument that I have not found?

Answer 1

Luigi has the concept of resources were you can limit the utilization of certain resources (in this case your server). You can configure that only a max number of task that require such resource to be executed concurrently. This is useful for example for databases where many tasks writing in parallel might hit server really bad.

Check the documentation:

http://luigi.readthedocs.io/en/stable/configuration.html?highlight=resources#resources

You basically add a section to your configuration:

   [resources]
   hive=2
   server=2

and then in your class

class MyTask(luigi.Task):
   resources = {'server': 1}

With this configuration you will only execute up two Task that use such resource. I know this is not what you were asking but I believe this is the closet feature Luigi has for your use case.