Search code examples
dbt

dbt pre-hooks cannot render/find macros?


A dbt_project.yml model pre-hook is failing to render macros. The following implementation:

# gold/dbt_project.yml

config-version: 2

...

models:
    gold:
        +pre-hooks: "{{ incremental_failsafe() }}"

And the following macro:

-- gold/macros/utils/incremental_failsafe.sql

{% macro incremental_failsafe() %}

{# /*
Used to wrap AND clauses that should only be run on dev/CI dbt runs.

For example, to limit the number of records scanned on an incremental
table. This macro will return True unless the target profile is 'prod':

`dbt run --target prod`
*/ #}

{% if target.name == "prod" %}
    {% set incremental_failsafe = False %}
{% else %}
    {% set incremental_failsafe = True %}
{% endif %}

{{ log("Running with incremental_failsafe: " ~ incremental_failsafe, info=True) }}
{{ return(incremental_failsafe) }}

{% endmacro %}

Results in the following:

(snowflake) Teghans-MacBook-Pro:gold tnightengale$ dbt compile
Running with dbt=0.17.2
Encountered an error:
Compilation Error
  Could not render {{ incremental_failsafe() }}: 'incremental_failsafe' is undefined

I know there was an issue with dbt 0.17.0 around pre-hooks as outlined in this slack exchange. I've tried using both +pre-hook: and pre-hook: specs in the yml as suggested in that conversation to no avail, despite the fact that I am running 0.17.2 (not 0.17.0). Any insight would be welcome!

EDIT: Solution

Truthfully, the answer is so dumb and obvious: the error resulted from having +pre-hooks: as opposed to +pre-hook:. However the suggested answer caused me to take a second look. I am marking it as correct because it gives a lot of helpful context on hooks in general. Cheers!


Solution

  • First impression - maybe you're mixing up where a model pre-hook for a particular model and a run pre-hook should go?

    Edit: Correcting myself on the above - Asker is attempting to perform model config pre-hooks, not run pre-hooks. I now believe that you need to move your pre-hook to an equal, prior level as your namespace configuation. See below:

    dbt_project.yml

    # A run pre-hook that applies to all runs would go here:
    on-run-start:
    - "{{ example_pre_run_macro() }}"
    
    
    models:
        # a pre-hooks to all models goes here *before* but equivalent depth to the namespace:
        +pre-hook: "{{ incremental_failsafe() }}"
     
        namespace:
            # Below configures models found in models/events/
            events:  
              +enabled: true
              +materialized: view
              # a pre-hook to a single model / model directory goes here:
              +pre-hook: "{{ example_pre_model_macro() }}"
    

    A really great example of how all this can be setup is from GitLab's dbt repo: Gitlab dbt_project.yml

    TLDR: Setup your dbt_project.yml like so

    # gold/dbt_project.yml
    
    config-version: 2
    
    ...
    
    models:
        +pre-hook: "{{ incremental_failsafe() }}"
        gold:
            <models config here>