Search code examples
kedro

Getting Kedro globals.yml to work with OmegaConfigLoader


I just started evaluating Kedro for use, and I began with a small project where I read data from a MS-SQL Server. The pipeline will run with a few months in between, and with new date ranges every time. To get this parameter (date) into the pipeline I looked into using globals.yml. The kedro run command works, I have created just one node that loads the data.

I load some parameters from globals.yml to use in catalog.yml.

  • If I use the standard lines in settings.py

    from kedro.config import TemplatedConfigLoader
    CONFIG_LOADER_CLASS = TemplatedConfigLoader
    
    CONFIG_LOADER_ARGS = {
                          "globals_pattern": "*globals.yml",
    }
    

    When doing this, it does run perfectly, BUT I get a warning message, specifically this

    FutureWarning: TemplatedConfigLoader
                          will be deprecated in Kedro 0.19. Please use the
                          OmegaConfigLoader instead. To consult the
                          documentation for OmegaConfigLoader, see here:
                          https://docs.kedro.org/en/stable/configuration/advanc
                          ed_configuration.html#omegaconfigloader
    
  • Perfectionist as I am, I want to get rid of this warning and be prepared for Kedro 0.19. Looking at the link in the warning, I replaced the above code in settings.py to

    from kedro.config import OmegaConfigLoader  # new import
    CONFIG_LOADER_CLASS = OmegaConfigLoader
    
    CONFIG_LOADER_ARGS = {
                         "globals_pattern": "*globals.yml",
    }
    

    Then it complains about __init__() got an unexpected keyword argument 'globals_pattern'.

  • OK so I try to remove the CONFIG_LOADER_ARGS thinking it is already there by default. Does not work either.

How do I get globals.yml to work properly with OmegaConf in Kedro?


Solution

  • This is now supported:

    If you are on 0.18.14, it's also supported in the 0.19.x versions: https://docs.kedro.org/en/0.18.14/configuration/advanced_configuration.html#how-to-use-global-variables-with-the-omegaconfigloader