I'd like to know if I can store the outuput of a DBT run (successes or failures, model name, execution time, maybe some "execution ID" into a table, so that I can do analytics of the successes / errors.
19:15:19 1 of 3 START table model dataset.table_1 ......... [RUN]
19:15:19 1 of 3 ERROR table model dataset.table_1 ......... [ERROR in 999s]
19:15:20 2 of 3 SKIP relation dataset.table................ [SKIP]
19:15:20 3 of 3 start table dataset.table_3................ [RUN]
19:15:20 3 of 3 OK created dataset.table_3................. [CREATE TABLE (1M rows, 10TB processed)]
Is there a way I can get this info tabulated? Maybe is there some package that does it?
model_name | started_at | finished_at | status
-----------------------------------------------------
dataset.table_3 | 19:15:20 | 19:15:20 | success
-----------------------------------------------------
dataset.table_2 | 19:15:19 | NULL | skip
-----------------------------------------------------
dataset.table_1 | 19:15:19 | 19:15:19 | error
I believe the best way to do this is to use the Brooklyn Data's excellent dbt-artifacts package. You will specifically be interested in the fct_dbt__model_executions table that it produces.
When dbt runs, it logs structured data to run_results.json and manifest.json files. This packages reads those files and then inserts data into your data warehouse based on their contents. This is much faster and better than adding model hooks that insert the data directly, at the end of each model execution.