Using hydra configuration management with python and yaml file , using notebook in databricks

Question

This is the sample code obtained from the document.

import hydra
from omegaconf import DictConfig, OmegaConf


@hydra.main(version_base=None, config_path=".", config_name="config")
def my_app(cfg: DictConfig) -> None:
    print(OmegaConf.to_yaml(cfg))



if __name__ == "__main__":
    my_app()

    db:
      name:groot
      driver: jdbc:mysql
      table: peer
      user: thanos
      password: goldfinger

in my Databricks workspace, I have a file named config.yaml in the same folder where this notebook is present. on running it, gives error - OverrideParseException: LexerNoViableAltException: 40473.

I am unable to understand what should I do to run it in Databricks.

I am trying to read configs from a config file kept in a directory using Hydra. all this is to be done in Databricks.

Answer 1

You use below approach.

Below is the config.yaml

Make sure you have correct syntax in config file.

from hydra import compose, initialize_config_dir
from omegaconf import OmegaConf

def main() -> None:
    with initialize_config_dir(version_base="1.3", config_dir="/Workspace/Users/path_to_config_folder/"):
        cfg = compose(
            config_name="config.yaml", return_hydra_config=True, overrides=[]
            )
        print(cfg.db)
        print(OmegaConf.to_yaml(cfg))
        
if  __name__ == "__main__":
    main()

Here, for config_dir give absolute path to the folder containing config.yaml.

For more info refer this page.