use luigi.DateParameter in a run function of a Luigi Orchestrator Task

Question

I have the following python code using the Luigi Orchestrator.

class AggregateArtists(luigi.Task):
    date = luigi.DateParameter(default=date.today() - timedelta(days=1))

    def requires(self):
        return []

    def run(self):
        ...

I want to use the date parameter in my run() function. The problem is that I don't know what type it is. In the doc, it seems that this parameter is a datetime.date, so I should be able to use the method self.date.strftime(). But this method is not avalaible for DateParameters.

My questions are:

• How can I use the variable date of my code within my run function? What type is it? A string, a datetime.date or something else?

• At some point, I need to convert this date to a string of the form YYYYMMDD, how can I do?

Answer 1

Your code is incomplete but I'll guess the rest is like the following. You must have an error somewhere because it works: DateParameter returns a value that is a python date. See luigi source code for details.

My tasks/foo.py:

from datetime import date, timedelta
import luigi


class AggregateArtists(luigi.Task):
    date = luigi.DateParameter(default=date.today() - timedelta(days=1))

    def output(self):
        return luigi.LocalTarget("/tmp/foobar.txt")

    def run(self):
        with self.output().open('w') as out_file:
            out_file.write(self.date.strftime("%Y%m%d") + "\n")


if __name__ == "__main__":
    luigi.run()

Running the task:

$ python tasks/foo.py AggregateArtists --local-scheduler

DEBUG: Checking if AggregateArtists(date=2015-12-03) is complete
INFO: Scheduled AggregateArtists(date=2015-12-03) (PENDING)
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 21831] Worker Worker(salt=179482616, workers=1, host=matagus-laptop, username=matagus, pid=21831) running   AggregateArtists(date=2015-12-03)
INFO: [pid 21831] Worker Worker(salt=179482616, workers=1, host=matagus-laptop, username=matagus, pid=21831) done      AggregateArtists(date=2015-12-03)
DEBUG: 1 running tasks, waiting for next task to finish
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: Worker Worker(salt=179482616, workers=1, host=matagus-laptop, username=matagus, pid=21831) was stopped. Shutting down Keep-Alive thread

Print the contents of the output file:

$ cat /tmp/foobar.txt 
20151203