Search code examples
yamljinja2dbt

env_var function in dbt project is not behaving as expected

in my dbt_project.yml

target-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}target" # directory which will store compiled SQL files
log-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}logs" # directory which will store compiled SQL files

I used env_var() function to use environment variables. If there is no DBT_RUNTIME_DIR set then the path should be just logs (because output from env_var should be empty string).

Unfortunately, it creates directory which is named {{ env_var('DBT_RUNTIME_DIR', '') }}logs which means function is not working as expected.

env_var_directory

If I change it to non-existing env-var, then it will not render:

log-path: "{{ env_var('non-existing-env-var') }}logs" # directory which will store compiled SQL files

11:17:06  Running with dbt=1.5.1
11:17:07  Encountered an error:
Parsing Error
  Env var required but not provided: 'non-existing-env-var'

so it means function is working, but not as supposed to.

If I add antything to "{{ env_var('DBT_RUNTIME_DIR') }}" it will just change the name of directory.

If I change to existing variable

export existing_variable=test
echo $existing_variable
>>> test

log-path: "{{ env_var('existing_variable') }}" # directory which will store compiled SQL files

existing_env_var_directory

Is there a bug in dbt or I missed something?

Solution

  • The Jinja context available to use inside dbt_project.yml is very limited. See https://docs.getdbt.com/reference/dbt-jinja-functions/dbt-project-yml-context. From that page, you should note the following:

    This applies to the models:, seeds:, and snapshots: keys in the dbt_project.yml file.

    Within those nodes in your yml, you're also limited to what Jinja methods you can call. Similarly, there is some Jinja available to you in on-run-start, on-run-end (https://docs.getdbt.com/reference/dbt-jinja-functions/on-run-end-context) and within your profiles.yml configuration as well (https://docs.getdbt.com/reference/dbt-jinja-functions/profiles-yml-context).

    The reason for the behavior you're seeing is because dbt_project.yml is used multiple times when running DBT. It's used for the path configurations at first, so when you have something like target-path: "{{ env_var('DBT_RUNTIME_DIR', '') }}target", it will just use the literal string. Then, it will do another pass to actually parse the Jinja in the project config file for the areas where it is allowed. However, it does parse the entire file during this phase which is why your version without the default values is failing.