Generate luigi config values at runtime (without adding them to parameters of tasks)

Question

So, I know that we can create configuration classes by extending luigi.Config, but is there a way to generate config values at runtime?

For instance, how would I do something like this:

def main():
    new_default_value = fetch_new_default_value()
    config = MyConfigClass()
    config.value = new_default_value
    luigi.build(MyTask())

And have config.value stay as new_default_value for the duration of execution? Also, is it possible to change it in the requires definition of a task, i.e., change set the value during execution?

Answer 1

How does config.value interact with MyTask? There's no reason you can't reference it in your Luigi task.

class MyTask(luigi.Task):

    def requires(self):
        new_default_value = fetch_new_default_value()
        return SomeOtherTask(param=new_default_value)

    def run(self):
        config = MyConfigClass()
        # do something that references config attributes

Update: another way to do it would be to pass in config as a parameter.

class MyTask(luigi.Task):

    config = luigi.DictParameter()

    def requires(self):
        return SomeOtherTask(param=self.config.value)

    def run(self):
        do_something(self.config.value)

config = MyConfigClass()
config.value = fetch_new_default_value()
job = MyTask(config=vars(config))  # assuming that vars() produces a dict
luigi.build([job])