Search code examples
pythonworkerluigidata-pipeline

Luigi not picking up next task to run, bunch of pending tasks left, no failed tasks

I am running a big Luigi workflow that is supposed run over a hundred tasks in total. The workflow goes well for quite a while, but at one stage it comes to a point where there are 15 pending tasks and all other tasks are successfully done, and no failed tasks. However, it doesn't seem to pick up any more of those pending tasks to execute. I have thoroughly looked at the logs and there are no errors. From that point on, it just periodically prints the following logs:

There are no more tasks to run at this time
There are 15 pending tasks possibly being run by other workers
There are 15 pending tasks unique to this worker
There are 15 pending tasks last scheduled by this worker

The luigi version that I am using is 2.6.1. And here is a screenshot:

enter image description here Any idea what might be going on here? Why would it think there is no task to run? Here is my luigi worker configuration:

[worker]
keep-alive = True
max-reschedules = 3
Solution

  • Figured out the issue with one of my co-worker's help. The problem was that of those 15 pending tasks, the tasks that are at the lowest level demand a bit of CPU resources (10 cores). However, I wasn't passing any luigi resource parameter when I launched my workflow. So even though there were tasks to run, luigi wasn't considering any of those as run-eligible because of lack of resources. Passing a resource parameter of {"cpu_cores": 10} solved my issue.

    Could also be done via luigi's configuration file like the following:

    [resources]
cpu_cores=10