Search code examples
python-3.6luigi

Luigi set config from within the code

I've wrapped a set of luigi task into a package. For now, each etl-task has it's own luigi.cfg in the same directory, however as all of those .cfg files are the same, it looks suboptimal. On top of that, I'd prefer to write S3 credentials from a different package/environmental var, rather than storing in flat files in git.

What is the best way to set default/general configuration for luigi from within package?

It looks like luigi.Config and luigi.interface.core packages are promising, but I am still not sure where I need to execute/initiate them.

Solution

  • You can try using

    luigi.configuration.get_config().set('core', 'some_key', 'some_value')

    in a running instance. You can access the same data using 

    luigi.configuration.get_config().get('core', 'some_key').