Search code examples
google-bigquerydbt

Updating a table in bigquery using DBT


I am trying to update a table in bigquery using DBT. The below command executes in bigquery;

Update {{ ref('my_table') }}
SET variable = 'value'
WHERE lower(variable) LIKE '%XX%' or lower(variable) like '%YY%'

However when I run it in DBT I get the following error

Server error: Database Error in rpc request (from remote system)
Syntax error: Expected end of input but got keyword LIMIT at [4:1]

Does anyone know why this is happening and how to resolve?


Solution

  • It's a little unintuitive at first I know but with dbt, every model is a select statement.

    You should instead think of doing something like:

    with cte as (
         select * from {{ ref('my_table') }}
         where <criteria>
    )
     select col1,
            col2,
            'value' as col3
     from cte
    

    Or possibly even simpler:

    SELECT 
      'value' as variable
    FROM {{ ref('my_table') }}
    WHERE lower(variable) LIKE '%XX%' or lower(variable) like '%YY%'
    

    Simply because during the dbt run cycle, the new values will be materialized into the new model.

    However, if you are looking for ways to clean underlying tables in a DRY way, I'd highly recommend this thread Modeling SQL Update Statements from the dbt discourse for some patterns on managing statements which handle specific value cleaning. Example from Kyle Ries:

    {% set mappings = {'something': 'boo', 'something-else': 'boo-else'} %}
    
    with source as (
            select * from {{ ref(‘stg_foobar’) }}
    ),
    
    final as ( 
    
            select
                case
                  {% for old, new in mappings %}
                    when other_column like ‘{{old}}’ then ‘{{new}}’
                  {% endfor %}
                end as column_name
            from
                source
    
    )
    
    select * from final