Search code examples
luigi

Luigi specify which config to use

I am wondering how one can specify which config file to use in a Luigi run from the command line as well as from the Python code.

I need something like this:

PYTHONPATH='.' luigi --module my_workflow MyTask --config=./myconfig.cfg  --local-scheduler

I am used to another pythonic wfms (snakemake). There I could specify the config file as a parameter. How can I achieve it in Luigi?

Solution

  • Usually, you can also specify using LUIGI_CONFIG_PATH, so you could just set the variable and start up a new process. I don't think you can set it at runtime unfortunately, or if you can it will be flaky. https://luigi.readthedocs.io/en/stable/configuration.html