A dbt_project.yml model pre-hook is failing to render macros. The following implementation:
# gold/dbt_project.yml
config-version: 2
...
models:
gold:
+pre-hooks: "{{ incremental_failsafe() }}"
And the following macro:
-- gold/macros/utils/incremental_failsafe.sql
{% macro incremental_failsafe() %}
{# /*
Used to wrap AND clauses that should only be run on dev/CI dbt runs.
For example, to limit the number of records scanned on an incremental
table. This macro will return True unless the target profile is 'prod':
`dbt run --target prod`
*/ #}
{% if target.name == "prod" %}
{% set incremental_failsafe = False %}
{% else %}
{% set incremental_failsafe = True %}
{% endif %}
{{ log("Running with incremental_failsafe: " ~ incremental_failsafe, info=True) }}
{{ return(incremental_failsafe) }}
{% endmacro %}
Results in the following:
(snowflake) Teghans-MacBook-Pro:gold tnightengale$ dbt compile
Running with dbt=0.17.2
Encountered an error:
Compilation Error
Could not render {{ incremental_failsafe() }}: 'incremental_failsafe' is undefined
I know there was an issue with dbt 0.17.0 around pre-hooks as outlined in this slack exchange. I've tried using both +pre-hook:
and pre-hook:
specs in the yml as suggested in that conversation to no avail, despite the fact that I am running 0.17.2 (not 0.17.0). Any insight would be welcome!
Truthfully, the answer is so dumb and obvious: the error resulted from having +pre-hooks:
as opposed to +pre-hook:
. However the suggested answer caused me to take a second look. I am marking it as correct because it gives a lot of helpful context on hooks in general. Cheers!
First impression - maybe you're mixing up where a model pre-hook for a particular model and a run pre-hook should go?
Edit: Correcting myself on the above - Asker is attempting to perform model config pre-hooks, not run pre-hooks. I now believe that you need to move your pre-hook to an equal, prior level as your namespace configuation. See below:
dbt_project.yml
# A run pre-hook that applies to all runs would go here:
on-run-start:
- "{{ example_pre_run_macro() }}"
models:
# a pre-hooks to all models goes here *before* but equivalent depth to the namespace:
+pre-hook: "{{ incremental_failsafe() }}"
namespace:
# Below configures models found in models/events/
events:
+enabled: true
+materialized: view
# a pre-hook to a single model / model directory goes here:
+pre-hook: "{{ example_pre_model_macro() }}"
A really great example of how all this can be setup is from GitLab's dbt repo: Gitlab dbt_project.yml
TLDR: Setup your dbt_project.yml like so
# gold/dbt_project.yml
config-version: 2
...
models:
+pre-hook: "{{ incremental_failsafe() }}"
gold:
<models config here>