How do you pass multiple arguments to a Luigi subtask?

Question

I have a Luigi task that requires a subtask. The subtask depends on parameters passed through by the parent task (i.e. the one that is doing the requireing). I know you can specify a parameter that the subtask can use by setting...

def requires(self):
    return subTask(some_parameter)

...then on the subtask, receiving the parameter by setting...

x = luigi.Parameter()

That only appears to let you pass through one parameter though. What is the best way to send through an arbitrary number of parameters, of whatever types I want? Really I want something like this:

class parentTask(luigi.Task):

    def requires(self):
        return subTask({'file_postfix': 'foo',
                        'file_content': 'bar'
        })

    def run(self):
        return


class subTask(luigi.Task):
    params = luigi.DictParameter()

    def output(self):
        return luigi.LocalTarget("file_{}.csv".format(self.params['file_postfix']))

    def run(self):
        with self.output().open('w') as f:
            f.write(self.params['file_content'])

As you can see I tried using luigi.DictParameter instead of a straight luigi.Parameter but I get TypeError: unhashable type: 'dict' from somewhere deep inside Luigi when I run the above.

Running Python 2.7.11, Luigi 2.1.1

Answer 1

What is the best way to send through an arbitrary number of parameters, of whatever types I want?

The best way is to use named parameters, e.g.,

#in requires
return MySampleSubTask(x=local_x, y=local_y)

class MySampleSubTask(luigi.Task):
    x = luigi.Parameter()
    y = luigi.Parameter()