Search code examples
luigi

Starting luigi programmatically and not waiting for the job result?


So the question is probably fairly simple. I have a job that is supposed to run for ~30 minutes and I don't want my program to wait 30 minutes for the result. I'd like to get a task name or id or something like that and return control to the user so that the user could later query the process state using my program.

https://github.com/spotify/luigi/blob/695010420105dc836096201a00d2f032f0143b84/luigi/interface.py

Looking at the interface.py I don't see anything relevant. Is it not possible? Or how do I achieve this? Thanks!

ps. If someone could point me at where to look for querying luigi daemon for a task status, i would be grateful.


Solution

  • I see a few questions here:

    1. How to get a task name or id: in the task, you can get this with self.task_id You can write this out to a file, a database, a queue etc. in your task's run method.
    2. How to return control to the user. Should be as simple as backgrounding the task (e.g., follow the luigi command with an & in *nix or hit ctrl-z and then type bg at the prompt.
    3. How to get the task's status- you can have it write out to something while running, but can also check the api at http://yourluigidhost:8082/api/task_list