Is there any way to access the kedro pipeline environment name? Actually below is my problem.
I am loading the config paths as below
conf_paths = ["conf/base", "conf/local"]
conf_loader = ConfigLoader(conf_paths)
parameters = conf_loader.get("parameters*", "parameters*/**")
catalog = conf_loader.get("catalog*")
But I have few environments like "conf/server"
, "conf/test"
etc, So if I have env name available I can add it to conf_paths as "conf/<env_name>"
so that kedro will read the files from the respective env folder.
But now if the env path is not added to conf_paths, the files are not being read by kedro even if i specify the env name while I run kedro like kedro run --env=server
I searched for all the docs but was not able to find any solution.
EDIT:
Elaborating more on the problem.
I am using the above-given parameters and catalog dicts in the nodes. I only have keys that are common for all runs in conf/base/parameters.yml
and the environment specific keys in conf/server/parameters.yml
but when i do kedro run --env=server
I am getting keyerror
which means the keys in conf/server/parameters.yml
is not available in the parameters dict. If I add conf/server
to config_paths kedro is running well without keyerror.
You don't need to define config paths, config loader etc unless you are trying to override something.
If you are using kedro 0.17.x, the hooks.py will look something like this.
Kedro will pass, base, local and the env you specified during runtime in conf_paths
into ConfigLoader
.
class ProjectHooks:
@hook_impl
def register_config_loader(
self, conf_paths: Iterable[str], env: str, extra_params: Dict[str, Any]
) -> ConfigLoader:
return ConfigLoader(conf_paths)
@hook_impl
def register_catalog(
self,
catalog: Optional[Dict[str, Dict[str, Any]]],
credentials: Dict[str, Dict[str, Any]],
load_versions: Dict[str, str],
save_version: str,
journal: Journal,
) -> DataCatalog:
return DataCatalog.from_config(
catalog, credentials, load_versions, save_version, journal
)
In question, I can see you have defined conf_paths
and conf_loader
and the env path is not present. So kedro will ignore the env passed during runtime.