Detect table to view materialization change in CI

Is there an easy way to detect a change in materialization in CI, to avoid a dbt run failure with the error

Compilation Error in model stores_stores
(models/marts/core/blah.sql)   Trying to create view
`blah`.`dbt`.`blah`, but it currently exists as a
table. Either drop `blah`.`dbt`.`blah` manually,
or run dbt with `--full-refresh` and dbt will drop it for you.

Thanks!

Solution

There is a way, however it's not particularly "easy".

What you can do is leverage the artifacts that dbt generates.

manifest.json: produced by compile, run, test, docs generate, ls
run_results.json: produced by run, test, seed, snapshot, docs generate
catalog.json: produced by docs generate

The information for materialisation change can be found in both the run_results and the manifest. However, in your context of adding a check to the CI in order to fail early, you want to be notified before getting an error from a dbt run. So you could actually generate the manifest.json with dbt compile.

In the nodes key of the manifest, each node will have a config.materialized key that you can look at. You can parse that with the command line or with python and store the result to a JSON file which holds the materialisation information of each model. That file can be checked-in into your code for example.

cat target/manifest.json | jq '.nodes | to_entries | map({node: .key, materialized: .value.config.materialized})' > old_state.json

Then after you've made a change to your dbt code, you need to run

dbt compile  # generates new manifest.json
cat target/manifest.json | jq '.nodes | to_entries | map({node: .key, materialized: .value.config.materialized})' > new_state.json

You can then compare two states with e.g. diff in the command line. I'll put an example output here:

$ diff old_state.json new_state.json
12c12
<     "materialized": "table"
---
>     "materialized": "view"

As I said, this is not "easy" per say, but I hope my answer gave you some ideas on how to proceed to get what you want. If you're interested in more details, you can check my blog post on the topic.